TPU training runs extremely slowly. 1 step takes 10x amount of time compared to single GPU

🐛 Bug
I ported my pytorch code to pytorch-lightning hoping to use TPU for speed up.  The ported pytorch-lightning  averages 2-3 steps per second on a single GPU. But when I run the exact same code on Colab TPU runtime and change  pl.Trainer(gpus=1)  to pl.Trainer(tpu_cores=8), a single step takes 10 - 20 seconds.
It is hard to make a simple notebook for reproducing the problem as the entire training script is tightly couple with an complex repository.  Also the steps to properly configure conda environment for training script to run is very complex.
Best I can do is to sharing the  pytorch-lightning code in a gist.
https://gist.github.com/NakedKoala/1a2fb843c08d493a3943d87e2b7010f0
Expected behavior
TPU training is not slower than single GPU training

I think we have fixed it on master, mind check latest pip install pytorch-lightning==1.3.0rc2
cc: @kaushikb11

I tried pip install pytorch-lightning==1.3.0rc2.  Did not help. Still taking half-minute per iteration.  Any earlier version ( before this bug was introduced ) that you can suggest ?

Hey @NakedKoala,
I looked at your gist. You are using only 1 tpu cores. Did you try 8 ?
Best,
T.C

Hey @NakedKoala,
I looked at your gist. You are using only 1 tpu cores. Did you try 8 ?
Best,
T.C

Yes. I started with 8 and saw the slowness and tried to debug by using 1 tpu (may be 1 tpu might avoid issues relating to spawning large number of processes )  But it did not speed up.    Even just 1 tpu should not be 10x slower than a single GPU

Yes, that's definitely annoying.
I didn't notice special slowness when trying Lightning Transformers on TPU. I think it was 3-5 times faster for me compared to 2 gpus training.
Mind putting together a reproducible script ?
Best,
T.C

Yes, that's definitely annoying.
I didn't notice special slowness when trying Lightning Transformers on TPU. I think it was 3-5 times faster for me compared to 2 gpus training.
Mind putting together a reproducible script ?
Best,
T.C

I will put together a public Colab within half a hour

Yes, that's definitely annoying.
I didn't notice special slowness when trying Lightning Transformers on TPU. I think it was 3-5 times faster for me compared to 2 gpus training.
Mind putting together a reproducible script ?
Best,
T.C

I will put together a public Colab within half a hour

Actually, do you mind sharing you gmail address. I will share my Private colab notebook with you. ( Repro notebook needs to fetch kaggle competition data.  Don't know how to do it without putting my Kaggle credential explicitly in the notebook )

Yes, that's definitely annoying.
I didn't notice special slowness when trying Lightning Transformers on TPU. I think it was 3-5 times faster for me compared to 2 gpus training.
Mind putting together a reproducible script ?
Best,
T.C

Working on repro notebook right now.
Q:  Do you have a Kaggle account ?

Yes, that's definitely annoying.
I didn't notice special slowness when trying Lightning Transformers on TPU. I think it was 3-5 times faster for me compared to 2 gpus training.
Mind putting together a reproducible script ?
Best,
T.C

Here is the reproduction notebook. Please follow the instruction contain within to reproduce.
Please save the files in the depdendency_files.zip into your colab runtime and copy them to the respective directory ( instruction is in the notebook).
Please let me know if you have trouble running the notebook
Colab:
https://colab.research.google.com/drive/1nVZ6OzyJRyTMouWJiTTyL3KbLcMvm0DY?usp=sharing
Dependency Files:
https://drive.google.com/file/d/1OhlaIRCEmxLMcfFrta99CVN7qL8-wkZs/view?usp=sharing

I used TPUSpawnPlugin in the Trainer. Here is the printout.  I am unable to decipher this.  Any info regarding the slowness can be recovered from below ?
pt-xla-profiler: TransferFromServerTime too frequent: 449 counts during 6 steps
pt-xla-profiler: Op(s) not lowered: aten::im2col, aten::im2col_backward,  Please open a GitHub issue with the above op lowering requests.

I also used import torch_xla.debug.metrics  to print metric after each training step.
Here is what gets printed out for 1 step
Metric: ExecuteTime
  TotalSamples: 210
  Accumulator: 03s913ms699.706us
  ValueRate: 010ms881.008us / second
  Rate: 0.712402 / second
  Percentiles: 1%=002ms114.601us; 5%=002ms277.371us; 10%=002ms464.266us; 20%=003ms687.511us; 50%=003ms131.265us; 80%=004ms029.872us; 90%=036ms142.378us; 95%=090ms578.779us; 99%=149ms292.477us
Metric: InboundData
  TotalSamples: 488
  Accumulator: 164.12MB
  ValueRate: 569.95KB / second
  Rate: 1.65502 / second
  Percentiles: 1%=0.00B; 5%=4.00B; 10%=96.00B; 20%=96.00B; 50%=1.50KB; 80%=576.00KB; 90%=1.15MB; 95%=2.25MB; 99%=2.25MB
Metric: InputOutputAliasCount
  TotalSamples: 17
  Accumulator: 12547.00
  ValueRate: 55.10 / second
  Rate: 0.0746605 / second
  Percentiles: 1%=1.00; 5%=1.00; 10%=1.00; 20%=1.00; 50%=1.00; 80%=1729.00; 90%=1729.00; 95%=1729.00; 99%=1729.00
Metric: IrValueTensorToXlaData
  TotalSamples: 464
  Accumulator: 03s319ms807.736us
  ValueRate: 011ms688.789us / second
  Rate: 1.49439 / second
  Percentiles: 1%=001ms383.500us; 5%=002ms592.581us; 10%=002ms682.237us; 20%=002ms869.295us; 50%=002ms161.842us; 80%=008ms008.658us; 90%=018ms629.230us; 95%=027ms902.053us; 99%=038ms516.177us
Metric: OutboundData
  TotalSamples: 505
  Accumulator: 335.15MB
  ValueRate: 1.08MB / second
  Rate: 1.62639 / second
  Percentiles: 1%=4.00B; 5%=4.00B; 10%=96.00B; 20%=96.00B; 50%=1.50KB; 80%=576.00KB; 90%=1.15MB; 95%=2.25MB; 99%=9.22MB
Metric: ReleaseDataHandlesTi
  Percentiles: 1%=001ms415.414us; 5%=002ms591.243us; 10%=002ms689.553us; 20%=002ms873.456us; 50%=002ms178.320us; 80%=008ms053.959us; 90%=021ms019.411us; 95%=027ms311.874us; 99%=194ms038.059us
Metric: TransferToServerTransformTime
  TotalSamples: 505
  Accumulator: 347ms810.469us
  ValueRate: 001ms116.931us / second
  Rate: 1.62639 / second
  Percentiles: 1%=103.857us; 5%=126.661us; 10%=137.395us; 20%=147.275us; 50%=210.807us; 80%=436.981us; 90%=919.946us; 95%=001ms075.260us; 99%=007ms229.038us
Counter: CachedCompile
  Value: 169
Counter: CreateCompileHandles
  Value: 41
Counter: CreateDataHandles
  Value: 13287
Counter: CreateXlaTensor
  Value: 59853
Counter: DestroyDataHandles
  Value: 11494
Counter: DestroyXlaTensor
  Value: 57239
Counter: DeviceDataCacheMiss
  Value: 27
Counter: MarkStep
  Value: 12
Counter: ReleaseDataHandles
  Value: 11494
Counter: UncachedCompile
  Value: 41
Counter: XRTAllocateFromTensor_Empty
  Value: 49
Counter: XrtCompile_Empty
  Value: 144
Counter: XrtExecuteChained_Empty
  Value: 144
Counter: XrtExec
  Value: 970
Counter: xla::fill_
  Value: 7
Counter: xla::gelu
  Value: 240
Counter: xla::gelu_backward
  Value: 168
Counter: xla::index_select
  Value: 10
Counter: xla::masked_fill_
  Value: 51
Counter: xla::mean
  Value: 6
Counter: xla::mm
  Value: 3936
Counter: xla::mul
  Value: 2759
Counter: xla::mul_
  Value: 6078
Counter: xla::native_batch_norm
  Value: 730
Counter: xla::native_batch_norm_backward
  Value: 511
Counter: xla::nll_loss_backward
  Value: 7
Counter: xla::nll_loss_forward
  Value: 8
Counter: xla::permute
  Value: 867
Counter: xla::relu
  Value: 30
Counter: xla::select
  Value: 1626
Counter: xla::slice
  Value: 3008
Coun
  Mean: 11s552ms531.926us
  StdDev: 20s626ms451.587us
  Rate: 0.280191 / second
  Percentiles: 25%=137.535us; 50%=044ms303.427us; 80%=18s625ms789.761us; 90%=37s382ms871.872us; 95%=01m03s260ms071.327us; 99%=01m07s957ms502.371us
Metric: XrtExecute
  TotalSamples: 3270
  Accumulator: 47s048ms508.525us
  Mean: 019ms094.347us
  StdDev: 055ms170.870us
  Rate: 3.80403 / second
  Percentiles: 25%=001ms375.058us; 50%=002ms682.463us; 80%=002ms444.386us; 90%=076ms055.945us; 95%=128ms289.253us; 99%=251ms442.059us
Metric: XrtExecutorEvict
  TotalSamples: 0
  Accumulator: nanB
  Mean: nanB
  StdDev: nanB
  Percentiles:
Metric: XrtReadLiteral
  TotalSamples: 7479
  Accumulator: 07s840ms184.193us
  Mean: 978.244us
  StdDev: 515.489us
  Rate: 3.87585 / second
  Percentiles: 25%=613.653us; 50%=812.726us; 80%=001ms296.944us; 90%=002ms883.298us; 95%=002ms024.700us; 99%=002ms425.886us
Metric: XrtReleaseAllocation
  TotalSamples: 4807
  Accumulator: 01s379ms343.721us
  Mean: 724.887us
  StdDev: 814.953us
  Rate: 4.02388 / second
  Percentiles: 25%=045.239us; 50%=370.298us; 80%=001ms385.083us; 90%=002ms110.651us; 95%=002ms454.791us; 99%=003ms918.296us

  TotalSamples: 505
  Accumulator: 16s433ms739.020us
  ValueRate: 054ms082.532us / second
  Rate: 1.66203 / second
  Percentiles: 1%=001ms154.548us; 5%=001ms263.647us; 10%=001ms352.982us; 20%=002ms520.668us; 50%=002ms939.772us; 80%=008ms871.668us; 90%=017ms787.944us; 95%=027ms694.975us; 99%=248ms443.830us
Metric: TransferToServerTransformTime
  TotalSamples: 505
  Accumulator: 326ms676.678us
  ValueRate: 001ms071.831us / second
  Rate: 1.662 / second
  Percentiles: 1%=075.977us; 5%=117.546us; 10%=127.075us; 20%=136.829us; 50%=184.361us; 80%=389.226us; 90%=926.046us; 95%=001ms102.299us; 99%=007ms001.112us
Counter: CachedCompile
  Value: 169
Counter: CreateCompileHandles
  Value: 41
Counter: CreateDataHandles
  Value: 13287
Counter: CreateXlaTensor
  Value: 59567
Counter: DestroyDataHandles
  Value: 11494
Counter: DestroyXlaTensor
  Value: 56953
Counter: DeviceDataCacheMiss
  Value: 27
Counter: MarkStep
  Value: 12
Counter: ReleaseDataHandles
  Value: 11494
Counter: UncachedCompile
  Value: 41
Counter: XRTAllocateFromTensor_Empty
  Value: 49
Counter: XrtCompil
  Accumulator: 02h48m51s024ms879.837us
  Mean: 11s643ms131.381us
  StdDev: 20s740ms733.091us
  Rate: 0.280356 / second
  Percentiles: 25%=138.210us; 50%=045ms777.329us; 80%=18s788ms563.419us; 90%=37s404ms787.324us; 95%=01m03s411ms989.704us; 99%=01m07s957ms502.371us
Metric: XrtExecute
  TotalSamples: 3272
  Accumulator: 47s181ms084.691us
  Mean: 019ms221.515us
  StdDev: 055ms277.790us
  Rate: 3.77203 / second
  Percentiles: 25%=001ms375.058us; 50%=002ms682.665us; 80%=002ms459.066us; 90%=076ms060.357us; 95%=128ms295.962us; 99%=251ms442.059us
Metric: XrtExecutorEvict
  TotalSamples: 0
  Accumulator: nanB
  Mean: nanB
  StdDev: nanB
  Percentiles:
Metric: XrtReadLiteral
  TotalSamples: 7480
  Accumulator: 07s842ms923.311us
  Mean: 979.176us
  StdDev: 516.001us
  Rate: 3.84252 / second
  Percentiles: 25%=613.653us; 50%=813.830us; 80%=001ms304.339us; 90%=002ms883.298us; 95%=002ms024.700us; 99%=002ms425.886us
Metric: XrtReleaseAllocation
  TotalSamples: 4826
  Accumulator: 01s404ms646.122us
  Mean: 740.004us
  StdDev: 824.547us
  Rate: 3.98847 / second
  Percentiles: 25%=044.948us; 50%=388.267us; 80%=001ms433.513us; 90%=002ms129.493us; 95%=002ms467.720us; 99%=003ms918.296us


[release-4.15] feat(api): Add ingress-controller-load-balancer-scope annotation

What this PR does / why we need it:
Manual cherry-pick of https://github.com/openshift/hypershift/pull/3848
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/retest-required

/retest-required

/retest-required

/lgtm

/jira cherry-pick HOSTEDCP-1519

/approve

/retest-required


update to go1.18, and general refactor / cleanup / fixes

Sorry for the massive number of commits / changes. This started as a bit of a fun "lunchtime" project to understand what this module does, and with the initial intent to cleanup some small things I spotted while reviewing https://github.com/moby/moby/pull/44458#pullrequestreview-1181617950, but it ended up in doing a bit more.
I split the changes into small commits, describing each step in the commit message, so it's best to review per-commit (and probably best to (if accepted) merge without squashing to keep the information in history 😅 🙏 .
If there's any changes in there that should be dropped; happy to do so 👍

@AdamKorcz ptal 🤗

Thank you for the PR! I will take a look over the weekend.

Thanks!!


Study the effect of compression algorithms on storage space required for offline products

What

Study the effect of compression algorithms on storage space required for offline products
https://docs.google.com/document/d/1l2gQXTbRwPd3RCLzL8B-QEAA30vmM0r4PX48bQbLK5M/edit
Graphs are not enough, but timing would be

Part of

#18

Findings in terms of time taken, I think we can go ahead with this
To write 25 products when searched jam when using compression
 Total extra time in milisecods to write: 72
To write 25 products when searched jam when not compressing
 Total extra time in milisecods to write: 74
Why it's bit fasterto write in case where we use compression?
Possibly when writing to sqlite the data to be written is less(cuz it's compressed)
so faster write speeds that componseates for the extra time of 3 miliseconds to compress
Decompress
On average to decompress a product json it took around 2 mili seconds
so if we show max 25 products in one go, the max time to decompress is around 50 mili seconds
Size comparison on how much storage used for both cases

With Compression around 0.8 megabytes for 100 products
Without Compression around 4.9 megabytes for 100 products

Tested all these in debug mode,

Emulator android 11
Cool(Le Eco) Phyical device

Results were the same for these 2 devices

Thank you @AshAman999, it really looks good to store compressed json instead of text: 5 times smaller, and no performance drawback in read or write.
For the record, you can't really start from a 2ms result for one product and extrapolate to 25 products. 2ms is approximately the smallest duration you can measure, it can mean 1.5ms or 2.49ms (a factor 2 between them), and sqflite may have somehow cached the query or the results. You could have tested reading 25 new products (or 10K). Well, anyway the general idea was that the added cost of read is insignificant, good!
And if possible, run your tests in profiler mode, that's precisely what the mode is here for.
Beyond this issue: let's say 1M products, it means 8Gb, right? A bit fat, isn't it?

Well, anyway the general idea was that the added cost of read is insignificant, good!

Yeah, that was the point here, had run it multiple times even reinstalling to run em seperately and had run in the profiler mode last time, had the same results I remember

Beyond this issue: let's say 1M products, it means 8Gb, right? A bit fat, isn't it?

Well we can store i guess around 10,000 products under 80mb, that's not bad , besides to let the user know if something is in the database or not here's the suggestion from pierre to just store the barcodes for 1M products if feasible

Store all barcodes available on OFF : so we can show the new product pipelineQuestions/Issues/Stuck
https://github.com/openfoodfacts/smooth-app/issues/2452#issue-1290482735

Well we can store i guess around 10,000 products under 80mb, that's not bad

I don't know what the target is, regarding the number of stored products.
If 10K is OK, I have good news for you: you can right away download and store 10K products to see if the app still works fine.

besides to let the user know if something is in the database or not here's the suggestion from pierre to just store the barcodes for 1M products if feasible

That would be in another specific table, with just the barcode, as primary key in TEXT. The barcode is about 13 character long, let's say it's UTF-8 then 26 bytes, additional table management let's say 35 bytes per product, therefore 35 Mb. Of course it would be way smaller if we assume that all (or most) products have numeric barcodes; we can store them as INT (6 bytes should be enough). That would be 15 Mb.

Of course it would be way smaller if we assume that all (or most) products have numeric barcodes; we can store them as INT (6 bytes should be enough). That would be 15 Mb.

My estimates were kinda same here as well(17mb) I thought of,
Besides https://fr.openfoodfacts.org/api/v2/search?fields=code&page_size=100 I don't know how else to get the barcodes for all the products and is there any other way to do so.
Besides tested ther apk with around 10,500 products cached, things were working just the same,
https://user-images.githubusercontent.com/57723319/177460608-3d3813cc-e79d-477a-8d48-bcdd8c046405.mp4

Besides tested ther apk with around 10,500 products cached, things were working just the same,

Cool!

Besides https://fr.openfoodfacts.org/api/v2/search?fields=code&page_size=100 I don't know how else to get the barcodes for all the products and is there any other way to do so.

Just use the page=2 parameter, and again and again :)
A bigger page size would help, too.
But that URL is good, I don't know a better one anyway.


Fix failing tests on Node 6.x

See https://github.com/sanity-io/block-content-to-hyperscript/pull/6#issuecomment-408681911 and the follow up comment.

Looks like this line fixed it: https://github.com/sanity-io/block-content-to-hyperscript/blob/19e1bdb7ea9f79b85a851d3ee1878bf41f3789a6/package.json#L56


Windows host support

Overview
As a user, I would like to run VMClarity on Windows hosts. For this, we need to:

[ ] provide families config example for running VMClarity on Windows
[ ] ensure that overall logic uses proper path traversal packages filepath
[ ] ops for block device management work on Windows

To scan a VM in VMClarity, we need to be able to list the block devices and mount/unmount these devices on the host machine.

List block devices is currently only supported for Linux and Darwin (https://github.com/openclarity/vmclarity/tree/main/utils/fsutils/blockdevice), the windows support needs to be added (e.g. https://github.com/jaypipes/ghw/tree/main/pkg/block)
Mount/unmount devices is currently only supported for Unix (https://github.com/openclarity/vmclarity/tree/main/utils/fsutils/mount), the windows support needs to be added (e.g. https://github.com/kubernetes/mount-utils)


specify chunksums by named args

Codecov Report

Merging #20 (c196ba7) into main (b7c4b7c) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main       #20   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            3         3
  Lines          227       229    +2
=========================================
+ Hits           227       229    +2

Impacted Files
Coverage Δ

chunkdup/chunkdiff.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Revert "[Issue 323] Fix a bug in DataWriter"

Reverts TextDB/textdb#324
This PR is not intended to be merged. It's now reverted for further reviews and discussions.

Checks failed because of Travis' error. The code has no problem.


Textures in glb model gets flipped

I have verified my glb model, it loads fine in any viewer.
But when i load it using jeeliz face filter. Textures in my glb model gets flipped.
What should i do? Here is my code, let me know if i am missing. Do i know need to flip canvas or some thing, i understand it is due to camera. texture gets reversed..??

const loader = new THREE.GLTFLoader();
loader.load( 'models3D/goggles.glb', function (glassesFramesGeometry) {
var model = glassesFramesGeometry.scene;
model.traverse ( ( o ) => {
if ( o.isMesh ) {
o.material.metalness = 1;
o.material.roughness = 1;
}
} );
threeGlasses.add(model);
});

This problem is related to THREE.js, not facefilter

Hi, i have already mentioned, The model loads fine using threejs. Texture are okay.

What version of THREE.js are you using? Did you tried with the latest one?

it is latest version

Please provide a full example so that I can test (source code with the GLTF model).

please find loader file in here
loader.txt
and the api
glb-jeeliz.txt

Hi... ! I have included source code and glb model in here inside zip file... please have a look

Provide the full project otherwise I won't look.

hi... !
I have included full source code...
siddharth.zip

Hi,
Your canvas is flipped horizontally because of CSS.
The CSS rule:  transform: rotateY(180deg); is applied.
So your canvas is like a real mirror and the texture seems to be flipped horizontally.
It is more realistic.
If you try to read text through a mirror it will look inverted too.
Best,
Xavier

Yeah... Thank u. I could get that it was due to mirror effect. I was unable to find this transformation line. Should have checked css file.

Hi can this face filter api could be made more stable. I was thinking if instead of live camera, if I could import a video file. Will that do?? If so how should I import a pre recorded video..?

A demo is provided in this repo using a video file. Plz check the readme.


duniverse build: update to latest versions of tools, remove odoc

odoc was in the class of 'this can never work', since of course
the tool depends on reading cmt files and therefore is dependent
on the precise compiler version.
This PR updates ocamlformat, duniverse and dune to the latest
versions as well. The next step in the duniverse tool is to do
a complete (and cache-friendly) bootstrap of ocaml and tools
from within duniverse install, so we should be able to remove
the hardcoded versions in the CI soon.

needs #169 to work correctly.


Translate blog/2023/01/06/unhealthy-pod-eviction-policy-for-pdbs/ into Japanese

This is a Feature Request

What would you like to be added

Translate blog/2023/01/06/unhealthy-pod-eviction-policy-for-pdbs/ into Japanese.
Why is this needed
This page has not been translated.
Comments

/language ja
/assign

/remove-lifecycle stale
/triage accept

/triage accepted


To Publish messages in announcement channels

Can we use the nodes to publish messages in announcement channels automatically?

Two ways for publish on announcement on 3.4.0-alpha.1:
Creating a new messaging and publishing it:
msg.payload = "Mi new announcement";
msg.channel = "1077588567209017384"; // Your channel ID
msg.crosspost = true;

return msg;

Publishing an existing message:
msg.channel = "1077588567209017384"; // Your channel ID
msg.message = "1077605124064682054";
msg.action = "crosspost";

return msg;


Ensure all feature flag based permissions are applied

There's a ff for create route that's unused, could be more

Fixed via #4195


Remove dead code in test_stats.py

I guess this was just mistakenly left here? cc @HyukjinKwon

Codecov Report

Merging #311 into master will increase coverage by 0.13%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #311      +/-   ##
==========================================
+ Coverage   93.59%   93.73%   +0.13%
==========================================
  Files          34       34
  Lines        3325     3318       -7
==========================================
- Hits         3112     3110       -2
+ Misses        213      208       -5

Impacted Files
Coverage Δ

databricks/koalas/tests/test_stats.py
100% <ø> (+8.33%)
:arrow_up:

databricks/koalas/__init__.py
88.88% <0%> (-3.71%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b775511...e4916a2. Read the comment docs.


Need to implement automatic CHANGELOG updates

This should link to issues/PRs, and generate author entries.

The releases page seems to track this reasonably well, closing


Redo animations for FPS performance

speaking with afbiklwnef njw;kfn jn.kj.h I get the impression that animations probably affect fps more than I anticipated. budhud animations were originally designed with a #base includes mindset, but now it seems that it may be performatively-inhibiting to have multiple animations within a single file. Along with this, I should probably remove the #base aspects of the animation (bh_ammo_base.txt, bh_health_base.txt, etc).
I'll need to do some benchmarks to verify that this is the case, though

half-assed benchmarks tests are suggesting animations are actually not the problem in this case, which is kinda sad because I was hoping it'd be that lmao. need to do more tests though

Honestly the fps issues, with a proper benchmarking demo, may be very marginal. Regardless, it may not be a bad idea to just redo my animation files to clean them up a bit.


Add github issue template

Expected Behavior
As contributer i wand a well-defined issue like https://github.com/stevemao/github-issue-templates/blob/master/simple/ISSUE_TEMPLATE.md
Steps to Reproduce the Problem

Create new issue on guthub

I can help you!

Hey :) big thx.
I want an scrum|userstory like issue template.
For example https://github.com/stevemao/github-issue-templates/blob/master/simple/ISSUE_TEMPLATE.md or https://github.com/TalAter/open-source-templates/blob/master/ISSUE_TEMPLATE.md


grpc: Use google.protobuf.Empty instead of api.Ack

The protobuf stdlib provides an Empty type, which could be used instead of custom api.Ack type

Don't forget to do
go get -v github.com/golang/protobuf/ptypes
and  import "github.com/golang/protobuf/ptypes/empty/empty.proto"; in your proto
If you want to use it

@AntoineRondelet This is builtin no? Looks like it's in the protobuf src

Done for ttn@refactor.

Also, you should not need to import the .proto like @AntoineRondelet commented. If your protobuf installation is correct, these protos should be included in the compiler, so you can just use import "google/protobuf/empty.proto";.

Done


List of libraries and API clients

In a similar vain to #464, I think it might be useful to make a list somewhere of libraries that implement the OSV schema or an OSV API client to prevent reinventing the wheel.
So far I've come across:

Rust

https://github.com/gcmurphy/osv

Go

https://pkg.go.dev/golang.org/x/vuln/osv
https://github.com/G-Rath/osv-detector

.NET

https://github.com/JamieMagee/osv.net (OSV.Schema and OSV.Client packages)

But it's not exhaustive, and a quick GitHub search for api.osv.dev shows more results

Thanks! This is a great point.
We already have a section at https://github.com/google/osv.dev#third-party-tools that we could expand on. Feel free to create a PR for this, or we can do this next week once we're back from a team event.

@andrewpollock do you wanna make this documentation change too?

Ack, I'll get onto this this week

I mentioned the wrong thing on https://github.com/google/osv.dev/pull/596, this is fixed now.
For the record, the search results also included a bunch of forks from the OSV repo, so it's artificially higher than reality.


Docs: Missing autoCorrect option for various rules

It seems that ktlint supports autoCorrect for indentation rule but it is not declared on detekt side.
Could it be added so that we can benefit from that? I didn't check other rules, maybe some others are missing as well.
edit: Seems to be missing for:

[ ] IndentationRule
[ ] AnnotationOnSeparateLine (AnnotationRule)
[ ] MaximumLineLength
[ ] NoWildcardImports
[ ] Filename
[ ] ImportOrdering

edit: it has actually not much impact, because it only allows you to disable/enable autocorrect per rule. But still it would be a nice docs benefit.


update the twitter-X logo #265

i update twitter older logo with X logo using font awesome
after : -

@ceilican @heychirag
can please review this PR


[CI] Cleanup travis and CI steps

Remove python 3.4 deprecated
Cleanup Django / python compatibility matrix

Hello @antonagestam, am I missing something here with the CI failing ?

@Djailla I'm scratching my head too, but haven't had time to look into it. Probably some issue with env vars in travis ...

Merged in #165


Setting to auto fix unhealty scheduled tasks in unattended mode

Is your feature request related to a problem? Please describe.
The wacs.exe is not always located at the same location when a new machine is automatically set-up / rebooted. This can't be easily fixed. We run wacs.exe once to install the cert and retup the scheduled task. Next reboot wacs.exe is called again, the path may be different then. Win-acme detects that an existing scheduled task is no longer pointing to the right exe, but it does not fix it.
Describe the solution you'd like
Option to auto fix the scheduled task if issues are found.
When running in attended mode you get the option to fix it. would be great if this would also be possible in unattended mode.
Describe alternatives you've considered
I could copy the wacs folder to a standard 'temp' directory on start-up, So that the task point to the right exe. But then I need to keep track of cleanup e.g. script that kind of stuff around it.

Hi @remyblok, running with the switch --setuptaskscheduler should do what you want already.

...And here I though I did my due diligence in reading the documentation. Great that it is already there! Thanks for your quick reply.

Hmm I've tested this, but it does not work.
wacs.exe --source manual --host "%CurrentHostName%"  --installation iis --installationsiteid<PHONE_NUMBER> --setuptaskscheduler

Output is:
 A simple Windows ACMEv2 client (WACS)
 Software version <IP_ADDRESS>9 (release, trimmed, standalone, 64-bit)
 Connecting to https://acme-v02.api.letsencrypt.org/...
 Scheduled task points to different location for .exe and/or working directory
 Scheduled task exists but does not look healthy
 Please report issues at https://github.com/win-acme/win-acme
 Running in mode: Unattended
 Source generated using plugin Manual: ***
 Overwriting previously created renewal

 Using cached certificate for [Manual] ***. To force a new request of the certificate within 1 days, run with the --force switch.
 Store with CertificateStore...
 Certificate with thumbprint A5B76B86B4B13B3E4F5790EA59E39B421C8F79D6 is already
 in the store
 Installing with IIS...
 No bindings have been changed
 Scheduled task points to different location for .exe and/or working directory
 Scheduled task exists but does not look healthy
 Proceeding with unhealthy scheduled task, automatic renewals may not work until
 this is addressed
 Next renewal scheduled at 2021/9/30 9:20:36
 Certificate [Manual] *** created

I went through the source code. Seems that I need to call wacs.exe twice. Once with the installation parameters and once with the --setuptaskscheduler. Then it works OK. Maybe needs a documentation clarification?
wacs.exe --source manual --host "%CurrentHostName%"  --installation iis --installationsiteid<PHONE_NUMBER>
wacs.exe --setuptaskscheduler

That's correct. It might be feasable to implement that parameter in the certificate creation process as well, I'll look into that.

Released in v2.1.19


Fluent method to specify API key (multi tenancy)

Hi guys
Is there a clean way to use multiple api keys with this package? Use case is a multi-tenant app where users supply their own keys
Overriding config values feels a bit hacky and doesn’t always work well with Octane.

Can always just use the underlying client https://github.com/openai-php/client

True. I’ll do that 👍

I'd rather rely on this package's default configuration than copy/pasting it into my Laravel app.
Is there any way to access the client object to call the withApiKey method?


delete .pyc files, more renamings and imports to make testing easier

1st things first, basic cleanup. .pyc files are autogenerated from .py files, no need to store them in the repo.

I didn't check that renaming .sage -> .py caused issues. Syntax of .sage files is slighly different from standard python...

OK, it works now with these renamings

IMHO it can be merged now - we'll do more later.

Oh this is wonderful, thank you!


weird freeplay thing

Describe your bug here.
freeplay doesn't save your score when the song is played in a custom difficulty
Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
No

It's a known "bug" and it's already planned to be fixed

ok then
i'll wait

Close this then

k


Use Session instead of os.environ

I've started work on refactoring all tools and code to rely more on the Session and avoid falling back to os.environ.
This way we can start introducing "live context switching" by changing the active session.

Refactored os.environ calls to api.Session
Removed AVALON_WORKDIR in favor of api.get_work_directory so it's easier to remain in sync with the current session. This could be added back in but seems like a (possibly redundant?) optimization

I want to discuss what else will be needed to make it manageable to easily switch a context.

Here we go, this starts making sense @mottosso!

Could you type up a summary of changes, along with an example of the task switching thingy?

Changes
This implements update_current_task and exposes it to the api.
With it you can change your current work environment (live in a Session) to a new asset, task or app. Let's assume we're working in asset Bruce and are in the modeling task within app maya.
Here's some steps we can take:
api.update_current_task(task="lighting")
# You'll now be in Bruce/modeling/maya

api.update_current_task(asset="Claire")
# You'll now be in Claire/modeling/maya

# Each call to update_current_task will trigger a `taskChanged` signal
# If you instantly want to switch to another asset on another task
# you should do
api.update_current_task(task="rigging", asset="Jack")
# You'll now be in Jack/rigging/maya

To make this work, this has changed additionally:

References to os.environ are replaced with the api.Session so it's always using the live session state. As such os.environ at some point (when updating the current task) could become out of sync with the Session. The os.environ describes the "launching environment" but the Session is the current live version of your current work environment, including the asset, task, app, etc.

The update_current_task emits a taskChanged signal that Avalon's default maya host will connect to and call "_set_project" to try and remain in sync with the current task - also when being updated, see: https://github.com/getavalon/core/pull/300/files#diff-a21b0383dbeefa22023f1e9517b9fb68R497

Looking good. Some minor nitpicks, then merge at will!

@BigRoy Think there are some mistakes in the example:
´´´python
api.update_current_task(task="lighting")
You'll now be in Bruce/lighting/maya
api.update_current_task(asset="Claire")
You'll now be in Claire/lighting/maya
Each call to update_current_task will trigger a taskChanged signal
If you instantly want to switch to another asset on another task
you should do
api.update_current_task(task="rigging", asset="Jack")
You'll now be in Jack/rigging/maya

How does it work when switching to a different app?

This is for switching within an app.

Ahh. Got confused by With it you can change your current work environment (live in a Session) to a new asset, task or app.

Actually, it would support switching to a different app context - I'm not sure yet though what that would mean in the context of running it in a host application. You can do update_current_task(app="nuke") for example - since the session would allow it and there's no other logic involved.
Note that app doesn't necessarily reflect the host, as all mayabatch, mayapy and maya are "app" maya - because that's what defines the application directory.

That should clean up things. 👍


Updated-readme

Updated README.md file by adding more information on the architecture to be used (MVVM Architecture).

Way to go 💯


[Docs] Release Notes for Meshery v0.5.1

An auto-generated pull request to document release notes for Meshery v0.5.1

Haha! @ramrodo :)

Can we use this as a template for the release of other adapters?

@leecalcote I'll close this PR as this not reflect the changes of Meshery v0.5.1. The next release should create the right PR for that.

Thank you, sir.
On Fri, Apr 2, 2021 at 12:48 PM Rodolfo Martínez Vega <
@.***> wrote:

@leecalcote https://github.com/leecalcote I'll close this PR as this
not reflect the changes of Meshery v0.5.1. The next release should create
the right PR for that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/layer5io/meshery/pull/2648#issuecomment-812637789,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABZYKEA24JNF2FU72W6HQPLTGX7NBANCNFSM42GO5TVQ
.


fix: install dep, add token for pushing commits

:tada: This PR is included in version 1.4.4 :tada:
The release is available on:

v1.4.4
GitHub release

Your semantic-release bot :package::rocket:


Update Kubernetes and Helm Versions

This commit updates K8s and Helm to the current stable release

This change is

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Create a community use page

Describe the solution you'd like
Recommended by @salaboy and I think its a great idea: we should have a page which shows different projects and their use of Sonobuoy. We can let folks submit their own projects just to be linked for visibility and knowledge sharing. Blog posts, plugins, CI scripts, etc.

100% this would be great to have.. as people can see how other projects are using sonobuoy for reference and reusing standard practices.

@johnSchnake In Knative, we just created an issue to track this: https://github.com/knative/community/issues/696 it might be a simple and quick starting point.

Sounds good; thats very simple too and wont require PRs. I'll be sure to link to it from the Community/docs pages when I can too

I feel like I'm going crazy; I could have sworn we had an issue like that a long time ago and I cant find it. Going to search a bit more and maybe will reopen it if I find it. Otherwise I'll recreate.

I shamelessly copied most of the post from Knative, I hope thats ok. I tweaked it to make it clear that I am OK with selfpromotion of sonobuoy related blog posts or scripts/plugins/etc.
Thanks @salaboy for the recommendation and resource


Bump to 1.6.0

@bors: r+

:pushpin: Commit afbd403 has been approved by alexcrichton

:hourglass: Testing commit afbd4034788bd6854bc22cf7ddf3ddb817718f08 with merge 19d4dbd65ba0bf252e192e599fddf57a7e7c24ac...

:sunny: Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 19d4dbd65ba0bf252e192e599fddf57a7e7c24ac to master...


Testing cases for multi-voice scores and more

Feature Description
Right now all test cases test scores with one voice. In order to ensure that several voices can be processed we need tests of more complex scores.
Solution Suggestion
The tests would not need to be tested with respect to their output, only the ability to parse the voices properly needs to be tested. This could be done either by manually constructing a multi-voice small score using the MX library and testing the NoteList directly in the test file, or creating a multi-voice score and testing the entire program on it using CMake.
Additional context
Multi-voice scores are not the only potential issue with these cases, it might be useful to have a complex test-case to make sure complexity in symbols in the score does not interfere with the parsing of the score into a NoteList.

Also, fix any issues where multi-voice structures cannot be properly parsed with the current solution.


Add .csv to gitignore

Standard formatting options in this program are yaml and csv. This
commit add all *.csv to gitignore. Generated files are not shown as
untracked changes anymore.

@lauripiispanen

Thanks for the contribution, @fabianschwarzfritz! I've merged it in my fork (see the commit referenced above).


hello world error

when I run python -m infinigen_examples.generate_nature --seed 0 --task coarse -g desert.gin simple.gin --output_folder outputs/hello_world/coarse
error like this:
you are using blender=4.0.0 which is not supported please use 3.6.0

Hello, could you share what installation method and logs you used?
Both our pyproject.toml (for pip installation) and interactive_blender.sh scripts explicitly specify to download bpy==3.6.0, so if they downloaded 4.0 somehow then something weird has gone on. If you installed bpy manually you may need to erase that installation and use our provided installation methods.

I changed bpy to 3.6.0, and run the  frist code successful, but when I run python -m infinigen_examples.generate_nature --seed 0 --task populate fine_terrain -g desert.gin simple.gin --input_folder outputs/hello_world/coarse --output_folder outputs/hello_world/fine
error like this:
[08:58:17.962] [infinigen.times] [INFO] | [meshing OpaqueTerrain]
[08:58:17.962] [infinigen.times] [INFO] | [OpaqueSphericalMesher: frontview_mesher]
[08:59:19.212] [infinigen.times] [INFO] | [OpaqueSphericalMesher: frontview_mesher] failed with <class 'AttributeError'>
[08:59:19.212] [infinigen.times] [INFO] | [meshing OpaqueTerrain] failed with <class 'AttributeError'>
[08:59:19.212] [infinigen.times] [INFO] | [MAIN TOTAL] failed with <class 'AttributeError'>
Traceback (most recent call last):
File "/root/anaconda3/envs/infinigen3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/anaconda3/envs/infinigen3/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen_examples/generate_nature.py", line 438, in
main(args)
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen_examples/generate_nature.py", line 409, in main
execute_tasks.main(
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen/core/execute_tasks.py", line 421, in main
execute_tasks(
File "/root/anaconda3/envs/infinigen3/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/root/anaconda3/envs/infinigen3/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.traceback) from None
File "/root/anaconda3/envs/infinigen3/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen/core/execute_tasks.py", line 344, in execute_tasks
terrain.fine_terrain(output_folder, optimize_terrain_diskusage=optimize_terrain_diskusage)
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen/terrain/core.py", line 266, in fine_terrain
fine_meshes, _ = self.export(spherical=True, cameras=[bpy.context.scene.camera])
File "/root/anaconda3/envs/infinigen3/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/root/anaconda3/envs/infinigen3/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
raise proxy.with_traceback(exception.traceback) from None
File "/root/anaconda3/envs/infinigen3/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen/terrain/core.py", line 123, in export
mesh = mesher([element for element in opaque_elements])
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen/terrain/mesher/spherical_mesher.py", line 108, in call
mesh1 = self.frontview_mesher(kernels)
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen/terrain/mesher/frontview_spherical_mesher.py", line 234, in call
catted_mesh = Mesh.cat([mesh, *meshes])
File "/media/huanqiu/e36b882f-b751-4f41-a460-3fa43a79ff32/infinigen3/infinigen/infinigen/terrain/utils/mesh.py", line 231, in cat
faces = np.zeros((0, 3), dtype=np.int)
File "/root/anaconda3/envs/infinigen3/lib/python3.10/site-packages/numpy/init.py", line 324, in getattr
raise AttributeError(former_attrs[attr])
AttributeError: module 'numpy' has no attribute 'int'.
np.int was a deprecated alias for the builtin int. To avoid this error in existing code, use int by itself. Doing this will not modify any behavior and is safe. When replacing np.int, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
In call to configurable 'export' (<function Terrain.export at 0x7fcbf557acb0>)
In call to configurable 'execute_tasks' (<function execute_tasks at 0x7fcbf51c6cb0>)
Error: Not freed memory blocks: 1, total unfreed memory 0.132236 MB
my first time install log:
logs2.txt
then I change async to async_(/OpenGL/GL/SGIX/async.py)
and second time install log:
logs3.txt


Bug: KeyError on s3 store ls command

Seems like there might be some form of degradation on the s3_store's ls method.
Traceback:
Traceback (most recent call last):
  File "polystore_test.py", line 16, in <module>
    print(store.ls(path='polyaxon-outputs'))
  File "/home/antonfr/.virtualenvs/polyaxon/lib/python3.6/site-packages/polystores/stores/manager.py", line 51, in ls
    return self.store.ls(path)
  File "/home/antonfr/.virtualenvs/polyaxon/lib/python3.6/site-packages/polystores/stores/s3_store.py", line 208, in ls
    return {'files': results['blobs'], 'dirs': results['prefixes']}
KeyError: 'blobs'

Here is the ls method that is called in S3Store. https://github.com/polyaxon/polystores/blob/master/polystores/stores/s3_store.py#L206
def ls(self, path):
    results = self.list(bucket_name=path)
    return {'files': results['blobs'], 'dirs': results['prefixes']}

However there is no concept of blobs in the case of s3 storage and results only contain the following keys: [keys, prefixes]
I see that both Azure and GCS have the concept of blobs in their stores which might have been the explanation for the error.
A simple fix would be to simply return results['keys'].
def ls(self, path):
    results = self.list(bucket_name=path)
-   return {'files': results['blobs'], 'dirs': results['prefixes']}
+   return {'files': results['keys'], 'dirs': results['prefixes']}

But this is probably not how the results are intended to be presented since the files are return as their entire path and no dirs are returned at all.
Here is a minimal example that I used together with a minio container to locate the bug.
import os
from polystores.stores.manager import StoreManager

data_path = 's3://polyaxon-outputs'
os.environ['AWS_ENDPOINT_URL'] = 'http://localhost:9000'
os.environ['AWS_ACCESS_KEY_ID'] = 'key'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'secret'
os.environ['AWS_REGION'] = 'us-east-1'
os.environ['POLYAXON_STORE'] = 's3'

store = StoreManager(path=data_path)

print(store.ls(path='polyaxon-outputs'))

yes the change suggested is correct!
Not sure about the keys though, I need to try as well.
I will fix the issue using the suggestion, and leave the keys for when I have time.

I think this is fixed. closing the issue.


run main.py find some error.

DEPRECATION: The OpenSSL being used by this python install (OpenSSL 1.0.2q  20 Nov 2018) does not meet the minimum supported version (>= OpenSSL 1.1.1) in order to support TLS 1.3 required by Cloudflare, You may encounter an unexpected Captcha or cloudflare 1020 blocks.
Traceback (most recent call last):
File "C:/Users/ayjin/Desktop/Develop/googleplay/main.py", line 1, in
from core.api import Api
File "C:\Users\ayjin\Desktop\Develop\googleplay\core_init_.py", line 1, in
from core.api import Api
File "C:\Users\ayjin\Desktop\Develop\googleplay\core\api.py", line 1, in
from core.scrap import Scraping
File "C:\Users\ayjin\Desktop\Develop\googleplay\core\scrap.py", line 12, in
class Scraping(object):
File "C:\Users\ayjin\Desktop\Develop\googleplay\core\scrap.py", line 26, in Scraping
def get_detail_search(self, urls: str | list) -> List[Dict]:
TypeError: unsupported operand type(s) for |: 'type' and 'type'

fine...when i switch another device the problem be solved.
maybe we can build this project toghter? could you leave a email we can contact?

DEPRECATION: The OpenSSL being used by this python install (OpenSSL 1.0.2q 20 Nov 2018) does not meet the minimum supported version (>= OpenSSL 1.1.1) in order to support TLS 1.3 required by Cloudflare, You may encounter an unexpected Captcha or cloudflare 1020 blocks. Traceback (most recent call last): File "C:/Users/ayjin/Desktop/Develop/googleplay/main.py", line 1, in from core.api import Api File "C:\Users\ayjin\Desktop\Develop\googleplay\core__init__.py", line 1, in from core.api import Api File "C:\Users\ayjin\Desktop\Develop\googleplay\core\api.py", line 1, in from core.scrap import Scraping File "C:\Users\ayjin\Desktop\Develop\googleplay\core\scrap.py", line 12, in class Scraping(object): File "C:\Users\ayjin\Desktop\Develop\googleplay\core\scrap.py", line 26, in Scraping def get_detail_search(self, urls: str | list) -> List[Dict]: TypeError: unsupported operand type(s) for |: 'type' and 'type'

i think you need upgrade your OpenSSL to lasted version

fine...when i switch another device the problem be solved. maybe we can build this project toghter? could you leave a email we can contact?

sure, you can fork this repo and just send pull requests or you can contact me at<EMAIL_ADDRESS>


Follow-up to k/website pr 29529, fix small nit for .NET

Follow-up to PR https://github.com/kubernetes/website/pull/29529
Addresses small nit: dotnet --> .NET in @celestehorgan 's comment
/assign @celestehorgan

/lgtm

/kind cleanup

Thanks @reylejano .
/approve


Contributing guide 1.x.x

Would be great to have a contributing guide for 1.x.x.

This would be great, would love to help.

I really appreciate that - I will be working on this very soon. Thanks!


Solve ESLint issues

Describe the changes you've made
Resolved the ESLint issues that were appearing after executing npm start. These warnings were appearing every time the command was executed.
Previous behaviour after executing npm start:
Compiled with warnings.

src/books.js
  Line 17:13:  Effect callbacks are synchronous to prevent race conditions. Put the async function inside:

useEffect(() => {
  async function fetchData() {
    // You can await here
    const response = await MyAPI.getData(someId);
    // ...
  }
  fetchData();
}, [someId]); // Or [] if effect doesn't need props or state

Learn more about data fetching with Hooks: https://reactjs.org/link/hooks-data-fetching  react-hooks/exhaustive-deps
  Line 84:15:  Redundant alt attribute. Screen-readers already announce `img` tags as an image. You don’t need to use the words `image`, `photo,` or `picture` (or any specified custom words) in the alt prop                                                                                jsx-a11y/img-redundant-alt

Search for the keywords to learn more about each warning.
To ignore, add // eslint-disable-next-line to the line before.

New behaviour:
Compiled successfully!

You can now view bookery-using-reactjs in the browser.

  Local:            http://localhost:3000
  On Your Network:  http://<IP_ADDRESS>:3000

Note that the development build is not optimized.
To create a production build, use npm run build.

Type of change
Please delete options that are not relevant.

[x] Removes warnings when starting the app

Checklist:

[x] I have performed a self-review of my own code.
[x] My changes generate no new warnings.
[x] New and existing unit tests pass locally with my changes.

@SantiagoJimenezQ Thank you for your contribution. Looking forward to more such amazing contributions 🥂


Remove remaining use of * imports.

@alex @glyph you can go ahead and take over the project, I'm not currently spending lots of time on the project


Wrong "Dernière synchronisation avec votre banque :" date

The date used on the "Rapports" page when resynching transaction is not lastChecked value of couchDB. Is it the expected behaviour?

Good catch! Thank you for all the bug reports, you're really helping me :+1:

Still not good when clicking "Synchronize now"

Just checked right now and it works locally. Can you force reloading (Ctrl Shift R or Ctrl F5) and retry, please?

On 0.5, the date is only updated after a reload of the page.

Duh, I never learn. Thanks.


Allow same folder imports (fixes #493)

Fixes #493
Done:

Clean up for test file - fixed lint errors and removed unnecessary symbols.
Added support for allow-siblings option.
Updated rule metadata to describe new option.
Updated README to mention new option.
(upd) Rule is no longer marked as typescriptOnly
checking for .. deep inside import path left out of scope of this rule

Questions:

at this moment rule allows paths like this 'absolute/path/…/path/file' which works with TS and Webpack. Does it make sense to prevent this kind of paths in all cases?

According to conversation in issue - arbitrary nesting is supported in relative paths. However it might lead for long relative import from files that are close to project root. May be it make sense to specify max allowed nesting for relative path? In our projects it is 2, if there is need for more - probably structure of components should be reconsidered.
For example option might be converted to something like  [true, {"allowed-relative-nesting-level": 2}] this will allow ./file, ./lvl1/file, ./lvl1/lvl2/file, but not ./lvl1/lvl2/lvl3/file. 0 will allow only files in the same folder.
What do you think?

Other problems/questions:

package-lock.json is updated "without reason". (Unable to install npm 5.7+ ATM to use npm ci) - updates not included in PR

recommended_ruleset.js contains updates not related to updated rule - updates not included in PR

Currently TS 3.0.1 is installed in this repo which leads to errors
Running "ts:test-data" (ts) task
Compiling...
Cleared fast compile cache for target: test-data
Using tsc v3.0.1
node_modules/typescript/lib/lib.dom.d.ts(17113,18): error TS2451: Cannot redeclare block-scoped variable 'setImmediate'.
node_modules/typescript/lib/lib.dom.d.ts(17114,18): error TS2451: Cannot redeclare block-scoped variable 'setImmediate'.
test-data/NoStringBasedSetImmediateTestInput.ts(3,15): error TS2451: Cannot redeclare block-scoped variable 'setImmediate'.

Commenting declare const setImmediate: (...args: any[]) => number; in later file helps.
Probably TS version should be updated in package.json

No lint for test files? Looks like there are a lot of violations (that are highlighted in editor), but no warnings/error with grunt all

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: IllusionMH sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

paths like this 'absolute/path/…/path/file'

If TS and Webpack allow them, it doesn't seem like it would make much sense to put extra effort in to block them. Redundant ..s in paths could be a separate lint rule. +1 to how you have it now.

Specify max allowed nesting for relative path

That's an interesting discussion. Would you mind holding off in this PR so we can discuss that in a separate issue filed once this goes in? Seems like that would be a bit trickier and it'd be nice to have this change in first.

recommended_ruleset.js contains updates not related to updated rule - updates not included in PR

I can take care of those separately, thanks for the heads up

Probably TS version should be updated in package.json

Indeed - I'll file a separate issue and push that through.

No lint for test files

Seems like that should be a new issue as well, agreed.

@JoshuaKGoldberg thanks for review and comments.

Specify max allowed nesting for relative path

That's an interesting discussion. Would you mind holding off in this PR so we can discuss that in a separate issue filed once this goes in? Seems like that would be a bit trickier and it'd be nice to have this change in first.

Yes, it can be discussed as a separate issue. I just wanted to know if it should be considered before merging this PR, because if  allow-siblings will be replaced by an object it might be a breaking change. Or it might be new rule strict-import-paths which will replacethis one.

Redundant ..s in paths could be a separate lint rule

At this point there is asymmetry in implementation - .. are not allowed at all when allow-siblings option is present (as was shown in initial issue), but can be used when no options passed (if not at the beginning of the sting).
Creating new rule will make allow-sibling options redundant, and users will have to decide between two conflicting rules no-relative-imports and no-parent-imports (suggested new rule) in their configuration which doesn't seems optimal.
Personally I lean towards forbidding .. in both cases (simple code change). In this case it will still be single rule that controls import paths, even if will do a bit more that name suggests (may be renamed later to more speaking strict-import-paths) and might find few errors in existing codebases.
Also I forgot to note that this rule is no longer marked as typescriptOnly which might find new errors in existing codebases, but looks logical.
P.S. Sorry for new long post that introduces more questions than answers.

..

I'm a little lost... Clarifying for my own sake: if the idea for the no-relative-imports rule is to ban relative imports, IMO that's all it should do. Starting with ../ should always be banned by the rule and starting with ./ should be banned by the rule if allow-siblings is not provided.
Whether there's a ../ somewhere deep within that folder path doesn't change the no-relative-import rule's logic, right? If that's correct, then this rule shouldn't check for that at all. Having a ../ deep within the path is just a quirk that could be managed elsewhere. If a new strict-import-paths rule gets added, that sounds fine, but for the scope of this rule it seems unrelated.
Am I missing something?

no longer marked as typescriptOnly

That's fine - I'll mark this as a breaking change in the release notes. With services such as languages and linters, it's fairly impossible to avoid breaking changes in minor or even patch versions. This change certainly warrants at least a minor version update and not just a patch.

Having a ../ deep within the path is just a quirk that could be managed elsewhere. If a new strict-import-paths rule gets added, that sounds fine, but for the scope of this rule it seems unrelated.

Agree. I've removed check for ../ deep inside path. Implementation and messages are only about start of the stings, while more strict checks will be considered for new rule (I think I will come up with finished rule proposal next week).
At this point I don't think that there are any open questions left for this rule and this PR can be merged if it passes review.

Awesome, thanks so much @IllusionMH! I appreciate you walking me through this 😊.
One miscellaneous thought on the encompassing rule: I personally do the opposite of this by always using relative imports except for aliased namespaces (e.g. using "@project/bundle-name" to refer to a Webpack bundle located at /src/BundleName)


Kotlin DSL IntelliJ script resolution miss configured environment variables

Kotlin DSL IntelliJ script dependencies resolution fail if the build fails to configure depending on the state of an environment variable configured in IntelliJ.
Expected Behavior
An environment variable configured in IntelliJ for Gradle is available in the build on Kotlin DSL IntelliJ script dependencies resolution.
Current Behavior
Environment variables configured in IntelliJ for Gradle aren't passed to the build for Kotlin DSL script dependencies resolution.
As a result, If the build fails to configure when a custom environment variable isn't set

Context
Original report https://youtrack.jetbrains.com/issue/KT-30974
A fix in Kotlin is required before a fix in Gradle can be made.
Steps to Reproduce

create build that fails to configure when a custom environment variable isn't set
import in IntelliJ
add the environment variable to the IntelliJ configuration
open a .gradle.kts script
observe an error message and inability to autocomplete in the editor as shown above

Your Environment
Gradle 5.4.1
IJ 2019.x with Kotlin 1.3.31

@eskatos The fix on the IDEA side will be shipped with Kotlin 1.3.40 (name of the environment key is gradleEnvironmentVariables)

Fixed in #10351, will ship in 6.0


Add HTML extension

This PR adds four new commands that give access to some HTML features: \htmlClass, \htmlId, \htmlStyle, and \htmlData.
Fixes #90.
Fixes #2029.

@ylemkimon this is great!

Does not work with fraction, eg:
\htmlId{id1}{a}+\frac{1}{\htmlId{id2}{b}}-c
id1 woks, but id2 does not.

@vd-man Both id1 and id2 seem to work here. Could you give more detail? Thank you.

This is ready for review!

Codecov Report

Merging #2082 into master will decrease coverage by 6.24%.
The diff coverage is 89.74%.

@@            Coverage Diff             @@
##           master    #2082      +/-   ##
==========================================
- Coverage   94.81%   88.56%   -6.25%
==========================================
  Files          81       82       +1
  Lines        5280     5318      +38
  Branches      926      932       +6
==========================================
- Hits         5006     4710     -296
- Misses        250      535     +285
- Partials       24       73      +49

Flag
Coverage Δ

#screenshotter
?

#test
88.56% <89.74%> (+0.02%)
:arrow_up:

Impacted Files
Coverage Δ

src/functions.js
100% <ø> (ø)
:arrow_up:

src/parseNode.js
84.21% <ø> (ø)
:arrow_up:

src/Settings.js
79.06% <ø> (ø)
:arrow_up:

contrib/render-a11y-string/render-a11y-string.js
89.88% <0%> (-0.36%)
:arrow_down:

src/buildHTML.js
95.07% <100%> (-0.71%)
:arrow_down:

src/functions/html.js
91.89% <91.89%> (ø)

src/functions/underline.js
16.66% <0%> (-83.34%)
:arrow_down:

src/functions/verb.js
17.64% <0%> (-82.36%)
:arrow_down:

src/functions/overline.js
23.07% <0%> (-76.93%)
:arrow_down:

src/functions/delimsizing.js
63.82% <0%> (-32.98%)
:arrow_down:

... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 634b4e2...363403f. Read the comment docs.

Would it be possible to add some docs for what \htmlId, \htmlStyle, \htmlClass and in particular \htmlData actually do? How can I preview the docs? If I click on "Deploy preview ready!" I get to a preview which has no updated HTML section in the docs.

@phlegmaticprogrammer see https://deploy-preview-2082--katex.netlify.com/docs/next/supported.html#html

Now it works for me! This is great progress, although some CSS properties seem to be still under full KaTeX control, for example in
A*\htmlStyle{color: red; font-family: Arial, Helvetica, sans-serif; font-style: normal; font-size:50px;}{A}
the color and font-size are applied, but the font-family and style are not. So a general \html would still be necessary I guess.

@phlegmaticprogrammer there's a ticket for adding \html, see #1596.

This seems to be merged, but not released yet. What is the timeline for having this in a version available for use? I would really like to use this feature soon


New UI components panel Issues

{NEW XX} Buttons doesn't work in foreign languages because use fixed english words to trigger. Should use keys.
In foreign languages, Traits list are duplicated because new elements use english words instead of its keys and each trait appears whice.

To clarify, the I think the issue is that {NEW text isn't translated, but still works, is that correct, or do they not work at all? (unable to test right now)


Fix PocketBaseUrl in Config.ts

This PR fixes an issue where the POCKETBASE_URL is evaluated incorrectly due to the ternary added for dev config. Currently it will evaluate the POCKETBASE_URL to either localhost or pocketbase and never to the actual value provided by env.PUBLIC_POCKETBASE_URL || getProcessEnvValue('PUBLIC_POCKETBASE_URL')

@goniszewski Any chance you could review this and merge (unless you have fixed the issue in another PR)


fixes #38

Fixes #38

Waiting for netlify preview.

Merging...

Closing, as completed.

@TechStudent11 Please read the current messege in #38 and see if you can add the thing in JS to see if the date is 25th and if it is p tag the text I added in this PR, a while ago.

Deleting branch, will restore or create a new PR, if @TechStudent11 can help with this.

Idk what to do, need @TechStudent11 's help.
NOTE: Deleting branch, will restore or create a new PR, if @TechStudent11 can help with this.


See dropdown/filter when I only can access one project

When on the team details page, if I only have one project, I still see a filter projects bar.

deprioritized, so closing.


fix timestep issue with generator fuel cost in subhourly data

Update Fuel Cost Calculation to Account for Time Step Duration
Changes

Modified the fuel cost calculation to correctly account for the time step duration.
Fuel costs are given per time step (e.g., per 30 minutes), not per hour.
To get the correct hourly cost, we need to divide the cost per time step by the number of hours in a time step: fuel_cost_per_time_step / hours_per_time_step
This conversion ensures we're using the correct hourly rate in our calculations.

Add a test for the subhourly fuel cost

Add a test for the subhourly fuel cost

I don't see the test. Can you also see what's going on with the failing Generator tests?

Add a test for the subhourly fuel cost

I don't see the test. Can you also see what's going on with the failing Generator tests?

I just put that there as a reminder. @Bill-Becker , i did just add tests for subhourly analysis to the offgrid case

I think we can close this and this PR replaces it (I also want to avoid those updates to the Project.toml and Manifest.toml - we should not be adding HiGHS.jl to the REopt.jl dependencies.
https://github.com/NREL/REopt.jl/pull/453

replaced PR with https://github.com/NREL/REopt.jl/pull/453


helper-grammar-regex-collection: don't capture quotes

As we had discussed elsewhere, this is a step towards simplifying
insert-link.js to not worry about quotes.

Looks good. Before merging, I would like to do some manual testing first ... just in case.

Ok, thanks!


GDK Pixbuf and GIMP plugins fail to decode valid JXL file

Describe the bug
See the attached JPEG XL file. It fails to load with GIMP, Eye of MATE, and ImageMagick. However, djxl, ffmpeg, and jxl-oxide all decode it correctly and report no errors.
To Reproduce
djxl george-smile4.jxl test.png
mpv george-smile4.jxl
gimp george-smile4.jxl
magick display george-smile4.jxl
eom george-smile4.jxl

Expected behavior
All libjxl-based decoders should decode the file correctly, as it is a valid file.
Screenshots

Environment

OS: Arch Linux x86_64
Compiler version: GCC 12.2.1
cjxl/djxl version string: JPEG XL decoder v0.9.0 6b1e5b03 [AVX2]

Attempt to open in eog on my computer:
./lib/jxl/modular/transform/transform.h:99: JXL_FAILURE: Invalid transform RCT type
./lib/jxl/fields.cc:450: JXL_RETURN_IF_ERROR code=1: visitor.Visit(fields)
./lib/jxl/modular/encoding/encoding.cc:458: JXL_RETURN_IF_ERROR code=1: status
./lib/jxl/modular/encoding/encoding.cc:597: JXL_RETURN_IF_ERROR code=1: dec_status
./lib/jxl/dec_modular.cc:413: JXL_FAILURE: Failed to decode modular DC group
./lib/jxl/dec_frame.cc:310: JXL_RETURN_IF_ERROR code=1: modular_frame_decoder_.DecodeVarDCTDC(dc_group_id, br, dec_state_)
./lib/jxl/dec_frame.cc:646: JXL_FAILURE: Error in DC group
./lib/jxl/decode.cc:1146: frame processing failed

But GIMP from flatpak install --user https://flathub.org/beta-repo/appstream/org.gimp.GIMP.flatpakref works:

I wonder if it's somehow related to the TOC permutation?

Update: GIMP release on windows can read the file fine, but the file-jxl plugin that's used by git main on linux fails, for me. Since the flatpak works as well, it looks like there might be a recent regression.


builtins checker is not running

$ ma run .
Could not find any problem related to decimal floating point (dfp) api
Could not find any problem related to integrated performance primitives (ipp) api
Looking for message passing interface (mpi) api problems in 1 suspect files.
Could not find any problem related to math kernel library (mkl) api
Looking for inline assembly problems in 1 suspect files.
Looking for char usage problems in 3 suspect files.
Could not find any problem related to hardware transactional memory (htm)
Could not find any problem related to long double usage
Could not find any problem related to performance degradation
Could not find any problem related to non portable pthread
Could not find any problem related to syscall usage
...

Note that it does not say:
Looking for x86-specific compiler built-in problems in 1 suspect files.

It also fails to run from the command line:
$ ma run -c builtins .

No migration reports found.

It will run as builtin:
$ ma run -c builtin .                                                                                [156/1951]
Looking for x86-specific compiler built-in problems in 1 suspect files.

================
Migration Report
================
Problem type: x86-specific compiler built-in
Problem description: x86 built-ins not supported in Power
...


Only Adding CodeCompletionItem without deleting existing one

NOTE: I know that questions should go on Stackoverflow, but the "Monaco editor" tag on stackoverflow seams quite inactive, so i will ask here.
languages#registerCompletionItemProvider will take the provided items if the returned array is not empty, else it will take the auto generated completion list.
My Question: How can i just append own items to the auto generated list?
Regards,
Liz3

this is
https://github.com/Microsoft/vscode/issues/21611

@jrieken this is not my problem, i want to be able to get the Word based suggestions in a way that when providing a own Completion item Provider i can add to the existing rather then replacing them completely

@liz3 I think completion item providers are additive. Did you actually try something out or is this a theoretical problem?

@alexandrudima Its simple, i am registering a own CompletionItemProvider, when the array i return has a length of 0(as result is empty) monaco falls back to the word based suggestion(as expected), if though the length is greater than 0 it will only provide the one from the array.
What i would like(in a perfect solution) is a additional parameter in: provideCompletionItems, which is an array containing the auto generated word based suggestions, making it possible to merge them into one big array.
Is there any other way to retrieve the word based suggestions code wise?

@liz3 But then your issue is the one @jrieken pointed to. You don't have a problem with another registered completion item provider, you have a problem with the built-in word suggestions, which disappear once you provide completions.
As a workaround, you can compute and provide word completions by yourself, it is not super complicated to split a string into words... You can always use the built-in implementation as reference - https://github.com/Microsoft/vscode/blob/095400ad6c5c6f2ac5ce4e9619d8de813e320dc5/src/vs/editor/common/services/editorSimpleWorker.ts#L150:L197


C++ class name not highlighted

VSCode Version: 1.26.1
OS Version: Ubuntu 18.04

Hello everyone,
I notice that class name with attributes preceding it are not highlighted when declared.
As you can see in this image, the EXPORT keyword is presumed to be the class name by syntax highlighter.

Does this issue occur when all extensions are disabled?: Yes

This issue was moved to textmate/c.tmbundle#62


PewPew improvements

Made several updates and fixes, including:

Fixed this issue: https://github.com/FamilySearch/pewpew/issues/263
Added a script called dev for running results-viewer-react; renamed the original dev script to dev-lint
Moved three YamlUrls subcomponents (RequestDetailsTab, HeadersView, and ResponseView) to their own files
The Edit Endpoint modal opens when "Add Endpoint" is clicked
The response now catches and displays CORS errors
The Response tab opens when "Test" is clicked

Looks good @blonde-mike! Let's git rid of the dev-lint and just add the lint command to the build script instead.


Add support for https proxy

Related to issue #33 this patch enables proxy support if a proxy URL is specified in the constructor to the Client class.

Have you tested establishing an RTM via the websocket with this?

No thanks for pointing that out.  I'll update the pull request when I have the change made and tested.

I have added support for configuring the WebSocket to proxy.  I've tested it locally and it is working.

Excellent thank you.  How do we go about getting a release to npmjs.org so I can finish the required patch for hubot-slack?


[grafana] Qoute ConfigMap label values in examples

What this PR does / why we need it:
Fix synax of ConfigMaps in examples for the Grafana chart.
Special notes for your reviewer:
Checklist

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md

Ping @davidkarlsen 🤗


fix: introspection bogus child paths

fixes #279

Thanks @mriemensberger
Changes LGTM.  I'll leave this for other reviewers for a few days and than do a release with the fix if there are no comments


Returned fields in response

Add an argument to request methods so it's possible to specify if response have all fields or just default (limited) ones.

done


Add site config to the list of deployments for status

Description
Site config was previously missing from the list of deployments to be checked in the status check
Related Issue
https://issues.redhat.com/browse/ACM-13731
Changes Made
Added site config the list of conditionals in the function ListDeploymentsForStatus

/lgtm


Create ublue-nix-install

I don't think this is the best repo for adding nix support, as it's primarily concerned with providing the drivers and support libraries for Nvidia hardware. I recommend making a PR in one of these repos instead:
https://github.com/ublue-os/main
https://github.com/ublue-os/fleek

I think for main/nvidia we want to keep these two sets of repos specifically more upstream-focused as we've been getting good feedback that a light touch is preferable there, that's part of why I'd like to codify it: https://github.com/orgs/ublue-os/discussions/125
(I'm fulling leveraging this in my downstream image though and I think the more downstream an image is the more opinionated it can be.)

yeah I think there should be a setup for nix ran by a command


Research reqs for twillo

Determine what the Twillo connection needs to work & and what languages/packages are supported, if necessary

Using python & flask
Using ngrok server hosting for testing & dev


attach-encrypt should support bucket sns notification

First step is adding sns support to mu.py

Alright, lessee here ...

I find attach-encrypt in code but not in docs.

I gather that the purpose of attach-encrypt is to install a Lambda (I guess that's the "attach" part?) that encrypts unencrypted s3 objects.

@kapilt So what's the tie-in to SNS? Are we publishing to SNS after encrypting? Or are we consuming from SNS as a trigger for encryption? My hunch is the former.

@whit537 - @kapilt can chime in, in case I'm missing something.  Just as the buckets can trigger lambdas, we want the ability to trigger SNS notifications.  In some edge cases, there are some buckets that already have lambdas attached for other valid reasons.  Since you can't have multiple lambda notifications on a single bucket, we want to trigger SNS instead.  From SNS, multiple lambdas can then be triggered.

consuming from sns, we already have publishing to sns.

First step is adding sns support to mu.py

Starting a PR for this in https://github.com/capitalone/cloud-custodian/pull/869 ...

Alright, the way I'm interpreting this ticket is that we want a new mode type sns that would look like:
policies:
  - name: s3-encryption
    resource: s3
    mode:
      type: sns
      topic: arn:aws:sns:region-0:123456789012:some-topic
    action: attach-encrypt

Is that the right track here, @kapilt?

for an initial pull request you can ignore the policy side, and just focus on mu.py support for a lambda event source on sns, there's several others there and then corresponding unit test in test_mu.py.
for a followup, the policy side here wouldn't be mode triggered, attach encrypt independently of the mode provisions a lambda on the bucket, and the logic would be in there depending on the extant bucket notification configuration on using a lambda notification or attaching to extant sns notification policies.

@kapilt So what's the end game here? Can you give me policy yaml or sample cli invocation demonstrating the final result we're going for here? Not to launch into full implementation but for context to understand the task on #869. Thanks.

for the use case around s3, the policy would just look like
policies:
  - name: s3-encryption
    resource: s3
    action: attach-encrypt

perhaps a higher bandwidth communication medium would be useful, let's try chat.

provisions a lambda on the bucket, and the logic would be in there depending on the extant bucket notification configuration on using a lambda notification or attaching to extant sns notification policies.

/me reads up on http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html ...

First step is adding sns support to mu.py

Ready in #869. Revisiting here ...

Benchmark: what happens when running the target policy currently?
policies:
  - name: s3-encryption
    resource: s3
    action: attach-encrypt

(cloud-custodian) $ custodian run -c custodian.yml -s .
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/bin/custodian", line 11, in <module>
    load_entry_point('c7n', 'console_scripts', 'custodian')()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/cli.py", line 319, in main
    command(options)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/commands.py", line 44, in _load_policies
    policies = policy_load(options, options.config)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/policy.py", line 62, in load
    raise Exception("Failed to validate on policy %s \n %s" % (errors[1], errors[0]))
Exception: Failed to validate on policy s3-encryption
 Additional properties are not allowed ('action' was unexpected)

Failed validating 'additionalProperties' in schema[78]['allOf'][0]:
    {'additionalProperties': False,
     'properties': {'actions': {'type': 'array'},
                    'comment': {'type': 'string'},
                    'comments': {'type': 'string'},
                    'description': {'type': 'string'},
                    'filters': {'type': 'array'},
                    'max-resources': {'type': 'integer'},
                    'mode': {'$ref': '#/definitions/policy-mode'},
                    'name': {'pattern': '^[A-z][A-z0-9]*(-[A-z0-9]*[A-z][A-z0-9]*)*$',
                             'type': 'string'},
                    'query': {'items': {'maxProperties': 1,
                                        'minProperties': 1,
                                        'type': 'object'},
                              'type': 'array'},
                    'region': {'type': 'string'},
                    'resource': {'type': 'string'},
                    'source': {'enum': ['describe', 'config']},
                    'tags': {'items': {'type': 'string'},
                             'type': 'array'}},
     'required': ['name', 'resource'],
     'type': 'object'}

On instance:
    {'action': 'attach-encrypt', 'name': 's3-encryption', 'resource': 's3'}
(cloud-custodian) $

policies:
  - name: s3-encryption
    resource: s3
    actions:
      - attach-encrypt

(cloud-custodian) $ custodian run -c custodian.yml -s .
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/bin/custodian", line 11, in <module>
    load_entry_point('c7n', 'console_scripts', 'custodian')()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/cli.py", line 319, in main
    command(options)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/commands.py", line 44, in _load_policies
    policies = policy_load(options, options.config)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/policy.py", line 63, in load
    return PolicyCollection(data, options)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/policy.py", line 76, in __init__
    Policy(p, options, session_factory=self.test_session_factory()))
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/policy.py", line 489, in __init__
    self.resource_manager = self.get_resource_manager()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/policy.py", line 586, in get_resource_manager
    return factory(self.ctx, self.data)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/resources/s3.py", line 92, in __init__
    super(S3, self).__init__(ctx, data)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/query.py", line 246, in __init__
    super(QueryResourceManager, self).__init__(data, options)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/manager.py", line 47, in __init__
    self.data.get('actions', []), self)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/actions.py", line 39, in parse
    results.append(self.factory(d, manager))
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/actions.py", line 58, in factory
    return action_class(data, manager).validate()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/resources/s3.py", line 612, in validate
    "attach-encrypt: role must be specified either "
ValueError: attach-encrypt: role must be specified either via assume or in config
(cloud-custodian) $

policies:
  - name: s3-encryption
    resource: s3
    actions:
      - type: attach-encrypt
        role: arn:aws:iam::644160558196:role/custodian-mu

(cloud-custodian) $ custodian run -c custodian.yml -s .
2017-02-07 05:55:57,947: custodian.policy:INFO Running policy s3-encryption resource: s3 region:default c7n:<IP_ADDRESS>
2017-02-07 05:56:02,404: custodian.policy:INFO policy: s3-encryption resource:s3 has count:8 time:4.46
2017-02-07 05:56:04,289: custodian.lambda:INFO Publishing custodian policy lambda function c7n-s3-encrypt
2017-02-07 05:56:10,408: custodian.lambda:INFO Publishing custodian policy lambda function c7n-s3-encrypt
2017-02-07 05:56:17,033: custodian.s3:ERROR Error attaching lambda-encrypt An error occurred (InvalidArgument) when callin
g the PutBucketNotificationConfiguration operation: The notification destination service region is not valid for the bucke
t location constraint
None
2017-02-07 05:56:22,897: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/policy.py", line 240, in run
    results = a.process(resources)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/resources/s3.py", line 665, in process
    results.append(f.result())
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/concurrent/futures/_base.py", line
 398, in result
    return self.__get_result()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/concurrent/futures/thread.py", lin
e 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/resources/s3.py", line 672, in process_bucket
    return source.add(func)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/mu.py", line 907, in add
    Bucket=self.bucket['Name'], NotificationConfiguration=notifies)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/botocore/client.py", line 251, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/botocore/client.py", line 537, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: The notification destination service region is not valid for the bucket location constraint
2017-02-07 05:56:22,901: custodian.commands:ERROR Error while executing policy s3-encryption, continuing
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/commands.py", line 141, in run
    policy()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/policy.py", line 572, in __call__
    return mode.run()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/policy.py", line 240, in run
    results = a.process(resources)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/resources/s3.py", line 665, in process
    results.append(f.result())
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/resources/s3.py", line 672, in process_bucket
    return source.add(func)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/mu.py", line 907, in add
    Bucket=self.bucket['Name'], NotificationConfiguration=notifies)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/botocore/client.py", line 251, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/lib/python2.7/site-packages/botocore/client.py", line 537, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: The notification destination service region is not valid for the bucket location constraint
(cloud-custodian) $

Success?
policies:
  - name: s3-encryption
    resource: s3
    filters:
      - Name: whit537-297
    actions:
      - type: attach-encrypt
        role: arn:aws:iam::644160558196:role/custodian-mu

(cloud-custodian) $ custodian run -c custodian.yml -s . --region=us-west-2
2017-02-07 06:04:27,152: custodian.policy:INFO Running policy s3-encryption resource: s3 region:us-west-2 c7n:<IP_ADDRESS>
2017-02-07 06:04:27,153: custodian.policy:INFO policy: s3-encryption resource:s3 has count:1 time:0.00
2017-02-07 06:04:29,979: custodian.lambda:INFO Bucket lambda notification present
2017-02-07 06:04:30,038: custodian.policy:INFO policy: s3-encryption action: attachlambdaencrypt resources: 1 execution_time: 2.88
(cloud-custodian) $

Alright, so what did that actually do? And what do we want to be done differently?

Bucket lambda notification present

My interpretation of https://github.com/capitalone/cloud-custodian/issues/297#issuecomment-275641802 is that we want to use an sns notification on the bucket instead of a lambda notification.

attach-encrypt should support bucket sns notification

Alright, I'm pretty sure this means we want to support the new SNSSubscription from #869 as an event source in addition to BucketNotification in s3crypt.

Moving to PR in #910 ...

@whit537 pls can we chill on the blow blow, it spams every project watcher, who probably don't want to line by line policy construction and validation errors.

re what it did, it provisioned a lambda function that will encrypt objects as they are added to the bucket. it did so by setting a bucket notification. extant buckets may already have a bucket notification in which case this won't work, per s3 bucket notification capabilities. @ocampocj i believe was looking in the case of an sns notification on a bucket to attach it.

D'oh! And here I thought I was leaving google-able traces for future confusees. 😜
I'll dial it back.

@kapilt Do you want automatic fallback to sns if a bucket notification lambda is already present? Or explicit events in the policy? Or both? Or ... ?
policies:
  - name: s3-encryption
    resource: s3
    filters:
      - Name: whit537-297
    actions:
      - type: attach-encrypt
        role: arn:aws:iam::644160558196:role/custodian-mu
        events: []

@ocampocj   for confirm on behavior ... so i think this gets a bit gray quickly, i think the request came as can we look at the existing notifications on the bucket and for topics already extant by path, and attach ourselves to them. if there is no notification set on the bucket we either want to continue as before with a lambda notify or create an sns topic for the bucket and attach ourselves.

note the event afaicr is slightly different which means ufuncs/s3crypt will need to disambiguate between the two styles (direct bucket invoke of lambda, and relayed through sns).

@kapilt after looking through some of the data we have, this is clearly becoming much more of an outlier scenario.  from an implementation point of view, I think we should be able to scan for existing SNS notifications and attach the lambda or create a new SNS notification and attach the lambda if one doesn't exist.

I think we should be able to scan for existing SNS notifications and attach the lambda or create a new SNS notification and attach the lambda if one doesn't exist.

Okay, this makes it sound like we are dropping bucket notifications entirely. Yes? Only SNS?

sorry @whit537, I was trying to expand on @kapilt 's previous comment but it came out strangely.  under normal circumstances, we should continue with the status quo and attach the lambda.  that would be the 'default' behavior.  however, we should also be able to specify in a policy that we would like to attach the lambda via SNS, in which case, we should be able to attach to an existing SNS or create one in the event that one doesn't already exist.

we should also be able to specify in a policy that we would like to attach the lambda via SNS

Any guidance on policy schema for this? Proposal:
policies:
  - name: s3-encryption
    resource: s3
    filters:
      - Name: whit537-297
    actions:
      - type: attach-encrypt
        role: arn:aws:iam::644160558196:role/custodian-mu
        attach_via: 'sns'

we should be able to attach to an existing SNS

How do we pick the existing topic? Any topic with `s3:ObjectCreate:*? A topic w/ an expected name? ... ?

example policy looks good.  though, if role is not specified, does it have a default?  role doesn't need to be specified when normally creating lambdas via policy, but is it needed for sns?
I think the lambda should attach to any existing sns if the topic already has 's3:ObjectCreate:*'.  otherwise, I think the policy (or policies) would get pretty complex having to cite different sns topic names.

role doesn't need to be specified when normally creating lambdas via policy, but is it needed for sns?

From what I can see, role does need to be specified for normal lambda creation. It's the role the Lambda runs as, so it's necessary in either case (unless I'm missing something?).

From what I can see

https://github.com/capitalone/cloud-custodian/issues/297#issuecomment-277964822

ok, makes sense.  it didn't look like it was a required field in the schema, but from your error it definitely seems so.
https://github.com/capitalone/cloud-custodian/blob/master/c7n/resources/s3.py#L639

it didn't look like it was a required field in the schema

Ah, good point. Maybe I can work that into this PR ...

Maybe I can work that into this PR ...

Actually, we already have a test for the ValueError that results from not providing role. My hunch is that we don't enforce role in the schema, because it can also be provided by --assume. 👍


Upgraded libssl version as previous one does not exists anymore and r…

…eturns 404.
Solves #247 and #246

fixed in master with tests. Thanks for the request!


it would be nice to have a stdlib-like hashtbl interface

Hi Hezekiah,
It would be nice to have a module which can be used as a drop-in replacement
for the stdlib Hashtbl module.
So that users can trivially switch from a small in-memory Hashtbl to a huge
on disk one.
Regards,
F.

So I did that recently:
https://github.com/UnixJunkie/phashtbl
But now I am looking for a backend that can handle 100M elements without any problem.


Update docs to include v3.7.2

[ ] Tests
[ ] Documentation
[ ] Release note

None required

Deploy preview for calico ready!
Built with commit a39599da3a247a412107dcea457c85a48f77d870
https://deploy-preview-2603--calico.netlify.com


Unable to "Download media files"

I am having an error downloading media files. See the error below. I am using the latest version, the commit 3 days ago for https. Maybe memrise changed more than just their https?
De-selecting the "download media files" options works perfectly still.
Traceback (most recent call last):
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/importer.py", line 146, in run
    course = self.memriseService.loadCourse(self.url, MemriseCourseLoader.Observer(self))
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/memrise.py", line 752, in loadCourse
    return courseLoader.loadCourse(self.getCourseIdFromUrl(url))
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/memrise.py", line 568, in loadCourse
    level = self.loadLevel(course, levelIndex)
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/memrise.py", line 659, in loadLevel
    self.notify('thingLoaded', thing)
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/memrise.py", line 549, in notify
    getattr(observer, signal)(*attrs, **kwargs)
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/importer.py", line 70, in thingLoaded
    self.downloadMedia(thing)
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/importer.py", line 51, in downloadMedia
    audio.localUrl = self.sender.download(audio.remoteUrl)
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/importer.py", line 110, in download
    return self.memriseService.downloadMedia(url, skipExisting=self.skipExistingMedia)
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/memrise.py", line 809, in downloadMedia
    data = self.openWithRetry(url).read()
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/memrise.py", line 715, in openWithRetry
    return self.opener.open(url)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/memrise.py", line 697, in https_open
    return self.do_open_wrapped(httplib.HTTPSConnection, req)
  File "/home/jasoons/Documents/Anki/addons/Memrise_Course_Importer/memrise.py", line 687, in do_open_wrapped
    response = self.do_open(http_class, req)
  File "/usr/lib/python2.7/urllib2.py", line 1195, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/share/anki/aqt/sync.py", line 441, in _incrementalSend
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 1278, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 353, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 601, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 838, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 272, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname u'static.memrise.com' doesn't match either of 'cloudfront.net', '*.cloudfront.net'

Well, unfortunately it's a lot of guesswork when it comes to memrise... the static.memrise.com urls seem to be plain http. Can you try again?

Great, thank you. It worked first time :)
I really appreciate the help!


Quick pick progress bar is not below input box

I just now noticed how the progress bar was a the very end of the quick pick but not below input box, I think it should be below input box because the progress is for what I typed.
Steps:

have some history entries in Cmd+P picker
be on a slow OS (Windows)
start to type and notice progress bar is below the history entries

//cc @daviddossett

Seems like this might have been a regression at some point. 👍 for relocating it back near the input.

I like where it is now

The reason I do not like it:

the progress bar is at a non-deterministic location depending on the number of items
the progress bar indicator is away from the location that triggers it (the input box)


Add PUT, PATCH, DELETE and cookie support

Apologies for the README formatting changes, I just realized my editor automatically made some header changes. Apart from that and the aforementioned new features, it also improves code coverage a bit and removes a couple of unused, unexported functions.

Thanks @mna! Sorry I forgot about this PR until today.
I made some adjustments here -- the main thing is that we can't rely on the FS test to check different methods like DELETE because http.FileServer has been changing how it responds to different methods in recent Go versions (see https://github.com/golang/go/issues/53501 and https://github.com/golang/go/issues/59470).


Update cassandra-driver-core to 3.9.0

Updates com.datastax.cassandra:cassandra-driver-core from 3.7.2 to 3.9.0.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.datastax.cassandra", artifactId = "cassandra-driver-core" } ]

labels: library-update, semver-minor

Superseded by #2103.