Ref docs issue

Error:

ERROR: Could not find a version that satisfies the requirement oyaml==0.9 (from versions: none)
ERROR: No matching distribution found for oyaml==0.9
WARNING: Location 'Please visit https://aka.ms/cfs for more details. If you need additional help, email (feedprotection@microsoft.com)./pip/' is ignored: it is either a non-existing path or lacks a specific scheme.
##[error]PowerShell exited with code '1'.

For more details check here:

https://apidrop.visualstudio.com/Content CI/_build/results?buildId=359815&view=results

@danieljurek for notification.

//cc: @JimSuplizio


Add info on methods sections; tense choice for papers.

@hannahbaumann Can you check these proposed edits to style guide and merge if it looks good?
cc  @Yoshanuikabundi in case he's interested though I don't think this affects his work.

@davidlmobley I don't think I have write access to this repository to merge the pull request


No detail info show on the list page when using Edge Agent, only get "No snapshot available".

I am not sure if it is a bug.
with Edge Agent installed, no detail info shows on the list page.

but it seems that the Edge Agent runs well. Any idea?

I've reopened this one since there's still a UX issue.
How you can force a snapshot being created is rather obscure.

I had the same issue, it must have been one of the following browser addons blocking, because after disabling them all together it started working immediately:
uBlock origin, i don't care about cookies, decentraleyes, Firefox built in protection

Has the same problem , portainer version is 2.13.1.
Can u please answer is it fixed in the latest version ?)

I am using the latest 2.15.1 BE edition. Have the same issue. Disabled uBlock and other tracking extensions of my browser. Still cant see detailed info. Even after clicking Refresh, it says "Success, Environments updated" and nothing happens.

+1
I face the same issue currently but did not invest time yet on searching the reason and a solution. I don't know yet if info will finally appear or not (2 Edge agents deployed few minutes ago)

Can this be reopened ?
I still have this issue :

@jamescarppe @samdulam can we reopen this ?


feat(cloud_firestore): Move Snapshot handling into a EventChannel

Description
As discussed in #4108, the current implementation for query snapshots (or document snapshots) is not working well in a multi isolate environment due to a static reference to the MethodChannel that calls back from Native to Dart.
This PR introduces a EventChannel per snapshot listener which works around this problem.
Related Issues
Closes #3671
Closes #4108
Closes #4267
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

Please help testing the PR
Add the following to your pubspec.yaml:
dependency_overrides:
  cloud_firestore:
    git:
      url: https://github.com/ened/flutterfire.git
      ref: fix/4108-cloud-firestore
      path: packages/cloud_firestore/cloud_firestore
  cloud_firestore_platform_interface:
    git:
      url: https://github.com/ened/flutterfire.git
      ref: fix/4108-cloud-firestore
      path: packages/cloud_firestore/cloud_firestore_platform_interface

Then run flutter clean && flutter pub get & test your iOS & Android Apps again.
Good test scenarios:

Schedule a piece of Firestore Work using flutter_workmanager
Update a Firestore document reference in the background_locator plugin callbacks

@Salakar this has spiralled quite a bit.
The channel reference is now transitioned to EventChannel for the 4 uses it had:

SnapshotInSync / Documents / Query Snapshot (easy stuff - one way communication once there is data)
Transaction handling 🤯. Lots of back & forth with the platform but seems to work in the sample project.

I have noticed there are no tests for snapshot cancellations (else, I'd have App crashes ;-)) so those should be added at some  point. The PR needs a lot more work, but please let me know if it's acceptable to work on above 4 components in the same PR. I prefer PRs to be smaller normally...

I have noticed there are no tests for snapshot cancellations (else, I'd have App crashes ;-)) so those should be added at some point.

Do you mean in the example app or in the e2e tests? I thought we had some in e2e; https://github.com/FirebaseExtended/flutterfire/tree/master/packages/cloud_firestore/cloud_firestore/example/test_driver - could be wrong though 😅 any additional tests you can add to e2e tests here would be extremely welcome!

The PR needs a lot more work, but please let me know if it's acceptable to work on above 4 components in the same PR. I prefer PRs to be smaller normally...

I don't mind either way. If you feel like you can break it down into smaller PRs that can function independently then be my guest - might make it easier on the review, otherwise a single PR is also fine - CI should cover most things before getting to review anyway.

Hey @ened
Thanks for starting this. I do have one question, since a lot of the current implementation was built on-top of the original code, we used MethodChannel pretty much everywhere. It was my understanding that using EventChannel is for sending one-way messages over the bridge, which wouldn't work with Transactions.
Originally, it was sending the result of a Transaction (anything the user returns) back to native, and then native sent that back to Dart land. The problem was, if the user sent a class or something to native you couldn't serialize it over the bridge & it'd throw an error. I noticed you've got a storeResult handler which seems to do this?
In the rework, I kept everything stored in Dart land (see here), and when the transaction finishes, I then pluck that value out and return it as the transaction result (see here). This way, it keeps away from native and the user can return whatever they like.
Sending and storing data on native works for primitive values like numbers and strings, but it'll break when trying to return something else, such as a DocumentReference.

Do you mean in the example app or in the e2e tests? I thought we had some in e2e; https://github.com/FirebaseExtended/flutterfire/tree/master/packages/cloud_firestore/cloud_firestore/example/test_driver - could be wrong though 😅 any additional tests you can add to e2e tests here would be extremely welcome!

Essentially, 2 snapshot observers (for the same query or document) should be opened simultaneously and the result of cancel awaited. Will need to double check this. There was definitely a ObjC exception that did not make it up. (or it was not raised, ObjC has some safeguards).

I don't mind either way. If you feel like you can break it down into smaller PRs that can function independently then be my guest - might make it easier on the review, otherwise a single PR is also fine - CI should cover most things before getting to review anyway.

A single PR it is. There is actually quite a bit of overlap on the Dart side which is better to read when in context. Thank you.

[...]
In the rework, I kept everything stored in Dart land (see here), and when the transaction finishes, I then pluck that value out and return it as the transaction result (see here). This way, it keeps away from native and the user can return whatever they like.
Sending and storing data on native works for primitive values like numbers and strings, but it'll break when trying to return something else, such as a DocumentReference.

For the non-native types Firestore utilises FirestoreMessageCodec which handles the conversion from Firestore objects like DocumentReferencePlatform to something Flutter can pass to native. There is a decoder on the native side as well, for example in FLTFirebaseFirestoreReader. I think that's a very cool way of utilising message codecs.  From my understanding it allows the plugin to pass those native objects around freely between Dart & Native; with automatic conversion.
In your examples only DocumentReference as a type was mentioned; as that type (and any other Firestore type) is covered, I really need to check if runTransaction results can contain other types (say my "SuperDuperObject" class):
I think it is not a problem, let's go through the code.

runTransaction is defined as template method with <T>: Therefore we expect any dart object to come back
runTransaction creates a StreamController: This means we need the result from that controller.
the controller starts the transaction by subscribing to the transaction stream with the correct parameters during onListen and keeping track in a separate subscription. The StreamHandler in iOS starts the transaction, then emits a single map value with a key named attempt
The Dart side picks up the attempt, generates the transaction command values and sends it back via storeResult.
Meanwhile, the iOS side has been awaiting the semaphore which signals a the transaction commands are ready and processes the commands.
The dart side moves on after storeResult is done and emits the result in the streamController.. I believe storeResult is a misleading name - suggestions welcome.

Therefore, the developer defined, non-firebase, objects never cross Dart <-> Native. Only the Firebase objects do.
There is a missing piece with transaction as the transaction here could still fail (NSError*) and the dart side would know nothing about it. I believe this is fixable by returning a appropriate value as storeResult response.

@ened Thanks - I'll put some time aside soon to review this however your comments generally make sense. Yeah the SuperDuperObject object problem did exist in the old code, so just wanted to make sure we didn't introduce regression.
I've not properly read through the code yet, only skimmed so I probably missed a few things.

@Ehesp the functionality is now somewhat testable, however there's a major issue left: I don't believe observing snapshots of more than 1 query or document simultaneously currently works properly. Instead of listening to on fixed EventChannel name, the plugin should provide on-demand channels (just like it's done with transactions now).
I'll sort that early next week along with a new e2e test.

Hi @Ehesp & @Salakar -
The tests are now extended and pass.
The mechanism has changed quite a bit from the initial PR.
Each document/query snapshot listener, in-sync listener or transaction will use their own EventChannel name & instance.
This is handled by a StreamHandler on the native side each which cleans up correctly when done.
The native plugins will also do house keeping and remove unused references.
I'd appreciate if you can do a review of the code and outline a path to merging it.

@russellwheatley if you get a chance would you mind taking a first pass at reviewing & testing this out? Thanks :)

@russellwheatley how are you doing - do you need help in reviewing this PR?

@ened Sorry for the merge conflicts with master, trying to get the null safety annotations in place which is no easy task ha

@Ehesp can you share an estimate when this PR would get reviewed / merged?
I'm happy to rebase once in a while, given there's a clear path to this being merged.

@russellwheatley thank you for reviewing, I'll address the feedback asap. Happy Christmas.

@russellwheatley this is now ready for a final review. I've reached out to a few users who reported issues with this part of the code in the past and hope they can confirm it's fixed. It certainly improved our main app. The web error seems to be CI specific, perhaps you can take a look into it? Thank you.

I may have actually discovered an issue with running this version on macos:
Undefined symbols for architecture x86_64:
  "_OBJC_CLASS_$_FLTDocumentSnapshotStreamHandler", referenced from:
      objc-class-ref in cloud_firestore(FLTFirebaseFirestorePlugin.o)
  "_OBJC_CLASS_$_FLTQuerySnapshotStreamHandler", referenced from:
      objc-class-ref in cloud_firestore(FLTFirebaseFirestorePlugin.o)
  "_OBJC_CLASS_$_FLTSnapshotsInSyncStreamHandler", referenced from:
      objc-class-ref in cloud_firestore(FLTFirebaseFirestorePlugin.o)
  "_OBJC_CLASS_$_FLTTransactionStreamHandler", referenced from:
      objc-class-ref in cloud_firestore(FLTFirebaseFirestorePlugin.o)
ld: symbol(s) not found for architecture x86_64

Could you check on your side?

I may have actually discovered an issue with building this version on macos:
Undefined symbols for architecture x86_64:
  "_OBJC_CLASS_$_FLTDocumentSnapshotStreamHandler", referenced from:
      objc-class-ref in cloud_firestore(FLTFirebaseFirestorePlugin.o)
  "_OBJC_CLASS_$_FLTQuerySnapshotStreamHandler", referenced from:
      objc-class-ref in cloud_firestore(FLTFirebaseFirestorePlugin.o)
  "_OBJC_CLASS_$_FLTSnapshotsInSyncStreamHandler", referenced from:
      objc-class-ref in cloud_firestore(FLTFirebaseFirestorePlugin.o)
  "_OBJC_CLASS_$_FLTTransactionStreamHandler", referenced from:
      objc-class-ref in cloud_firestore(FLTFirebaseFirestorePlugin.o)
ld: symbol(s) not found for architecture x86_64

Could you check on your side?

Did you run pod install after switching to this fork?
As you can see in the build logs, the macOS example builds on CI.

I have now (and flutter clean) but still seeing the same issue:
  firebase_core: ^0.5.3
  firebase_storage: ^5.2.0
  cloud_firestore:
    path: /Users/pschuegr/dev/flutterfire/packages/cloud_firestore/cloud_firestore
  firebase_auth: ^0.18.0

dependency_overrides:
  cloud_firestore:
    git:
      url: https://github.com/ened/flutterfire.git
      ref: fix/4108-cloud-firestore
      path: packages/cloud_firestore/cloud_firestore
  cloud_firestore_platform_interface:
    git:
      url: https://github.com/ened/flutterfire.git
      ref: fix/4108-cloud-firestore
      path: packages/cloud_firestore/cloud_firestore_platform_interface

@ened So just getting my head around the reasoning for actually using an Event Channel:

(Dart) The MethodChannel instance is exposed as a static.
(Native) When an event (from a collection/document subscription) is triggered, it was previously sending an event back via the MethodChannel.
(Dart) If the request was made via an isolate, when it comes back from native it can't access the MethodChannel instance

To confirm: This PR creates an EventChannel per request on the fly, so if it's in an isolate the EventChannel is created from an isolate?
Question: The reason this isn't an issue for calls such as get is because the MethodChannel returns a Future, and doesn't need access to events?

@ened So just getting my head around the reasoning for actually using an Event Channel:

(Dart) The MethodChannel instance is exposed as a static.

"static" meaning here that the native plugin had a static instance of the very same MethodChannel.
Also, the iOS plugin would previously overwrite that static variable every time it is instantiated (which is more than once in multi-isolate apps).

(Native) When an event (from a collection/document subscription) is triggered, it was previously sending an event back via the MethodChannel.
(Dart) If the request was made via an isolate, when it comes back from native it can't access the MethodChannel instance

To confirm: This PR creates an EventChannel per request on the fly, so if it's in an isolate the EventChannel is created from an isolate?

I'd rephrase to "This PR creates an EventChannel per request on the fly, so that the native side does not need to keep a record of a communication back channel which but instead exposes them on the fly".

Question: The reason this isn't an issue for calls such as get is because the MethodChannel returns a Future, and doesn't need access to events?

Essentially, the native side registers a method channel, it implements the onMethodCall method, which has access to the result object and can return values over the same wire, if you will.
This is not working for streams, which can generate zero, one or many elements at any time (as long as they're subscribed).
Also worth mentioning is there used to be a lot of book keeping like this: https://github.com/FirebaseExtended/flutterfire/pull/4209/files#diff-3fde93986bda31f622bde969df19fd5ace5349d19f3f7cf3c4265b502f35f27dL100. Moving that to native means that no 2 isolates could ever accidentally access to the same handle. (each dart plugin's isolate instance starts from 0).

@Ehesp refer to this ticket as well: https://github.com/FirebaseExtended/flutterfire/issues/4416.
The same underlying problem: The native plugin holds on to the channel and eventually crashes when it's gone. The quick fix is to add a null pointer check.
However, the proper check is to move the ID token listener https://github.com/FirebaseExtended/flutterfire/blob/master/packages/firebase_auth/firebase_auth/android/src/main/java/io/flutter/plugins/firebase/auth/FlutterFirebaseAuthPlugin.java#L528 into a EventChannel and only actually subscribe to ID token changes when the developer requested it.
This would also reduce memory usage (our App runs 2 full permanent isolates for example, only one of them sometimes needs access to FB auth, and never the ID token for example).

@ened Great thanks for explaining.
I can take a look at the auth side of things. I've also started work on the RTDB rework and have moved the query listener streams into event channels, following this PR as an example.

I can take a look at the auth side of things.

Do you have a estimate? We received crash reports so it may me on my list for next week.

I can take a look at the auth side of things.

Do you have a estimate? We received crash reports so it may me on my list for next week.

If you can look at Auth also that would be super helpful - our plate is quite full at the moment so any help with PRs is appreciated. @Ehesp is mostly out of office now for a few weeks.

cc @rrousselGit also for a final review if you can please. My quick glance and this LGTM (though not fully reviewed). With @russellwheatley & @rrousselGit approval I'm happy to ship

I can take a look at the auth side of things.

Do you have a estimate? We received crash reports so it may me on my list for next week.

If you can look at Auth also that would be super helpful - our plate is quite full at the moment so any help with PRs is appreciated. @Ehesp is mostly out of office now for a few weeks.

cc @rrousselGit also for a final review if you can please. My quick glance and this LGTM (though not fully reviewed). With @russellwheatley & @rrousselGit approval I'm happy to ship

hi @rrousselGit thank you for reviewing. The code has been updated & a major part of the review eliminated by deleting the code in question. ;-)). Please let me know if you have further comments.

hi @rrousselGit thank you for reviewing. The code has been updated & a major part of the review eliminated by deleting the code in question. ;-)). Please let me know if you have further comments.

Providing @rrousselGit is happy with the feedback changes I'd like to get this merged today 🙃

Providing @rrousselGit is happy with the feedback changes I'd like to get this merged today 🙃

@rrousselGit I had a look at forwarding the stack in method_channel_document_reference, however it is empty on the source if you will:
        }, onError: (error, stack) {
          print('add stack: $stack');
          controller.addError(convertPlatformException(error), stack);
        });

outputs ''
From scanning the native code, I can not see a way to push a custom stack trace either.
Therefore I'd say we ignore it for now.

@rrousselGit I had a look at forwarding the stack in method_channel_document_reference, however it is empty on the source if you will:
        }, onError: (error, stack) {
          print('add stack: $stack');
          controller.addError(convertPlatformException(error), stack);
        });

outputs ''
From scanning the native code, I can not see a way to push a custom stack trace either.
Therefore I'd say we ignore it for now.

Fair enough

Fair enough

After that we need to figure out a plan to fix the other plugins 🙈

@Salakar I'm looking at firebase_auth currently - do let me know if there is another channel to discuss / coordinate.

After that we need to figure out a plan to fix the other plugins 🙈

@Salakar I'm looking at firebase_auth currently - do let me know if there is another channel to discuss / coordinate.

Drop me a mail and I can invite you to our internal slack if you'd like. mike [at] invertase.io

Drop me a mail and I can invite you to our internal slack if you'd like. mike [at] invertase.io

any idea when this will be available on pub.dev? I'm excited to see if this fixes a lot of crashes

any idea when this will be available on pub.dev? I'm excited to see if this fixes a lot of crashes


Nest @minify inside another @minify

Hi,
would be great if you can nest the @minify directive inside another @minify directive, as below:
some.blade.php
@minify('html')

Yayyyy!

@endminify

main.blade.php

Thanks!

Quoting @minify documentation.

There aren't many valid use-cases for using this directive. Check out the alternatives at the bottom of the page first!


Estonian regional settings

Hey, when coming Estonia regional setting support ?
Thanks fro reply.

If you would like to contribute to translating ONLYOFFICE to Estonian, contact us at<EMAIL_ADDRESS>


Print preview always shows responsive tabs.

Regardless of viewport width tabs are always converted to accordion in print preview. This is in both chrome and ie. To reporduce simply look at the print preview of your own github index page.

That's because your printer makes it small enough that it switches. If you print on larger paper it would still be in tabs. It would be nice to have it fixed for printers, but I don't use this project, so I'm not going to be fixing it.


NEW LAB REQUEST: Proactive Bot

Add a new lab showing how to notify users using a proactive bot

Moved to new repo


Unit tests fail with latest version of seaborn

Description
Using the seaborn==0.11 makes scanpy/tests/test_plotting.py:test_violin and scanpy/tests/notebooks/test_pbmc3k.py fail. The warning
UserWarning: Vertical orientation ignored with only `x` specified.

is thrown by seaborn in the stack trace for the failed tests.
The problem seems to originate from here and here, where x and y are not both specified.
See #1417 and especially this comment by @fidelram for more details.

@WeilerP Any idea on how to solve the problem? I thought that setting orient='vertical' was going to solve the issue but we already do that.

I skimmed through the code only briefly yesterday but I think the problem might be that both x and y need to be specified: The error message seems to indicate so and the two lines seemingly responsible for the problem are the only ones where x and y are not both specified.
I'll have a closer look at it in a bit.

Looking at this function which is called by seaborn violin (https://github.com/mwaskom/seaborn/blob/cf8af7e1c4ffd5d0044d13323731fa5066c057c8/seaborn/_core.py#L1260) it seems that what we need to do is swap x and y.

Yes, I saw that too but I was hoping it could be prevented as it seems suboptimal to me and is probably confusing when looking at the code.
Is there a specific reason for using seaborn.FacetGrid instead of seaborn.catplot (see here)? Replacing the lines
# ...
kwds.setdefault('cut', 0)
kwds.setdefault('inner')

if multi_panel and groupby is None and len(ys) == 1:
    # This is a quick and dirty way for adapting scales across several
    # keys if groupby is None.
    y = ys[0]
    g = sns.FacetGrid(obs_tidy, col=x, col_order=keys, sharey=False)

    # don't really know why this gives a warning without passing `order`
    g = g.map(sns.violinplot, y, orient='vertical', scale=scale, order=keys, **kwds)

here by
if multi_panel and groupby is None and len(ys) == 1:
    g = sns.catplot(y=y, data=obs_tidy, kind="violin", col=x, col_order=keys, sharey=False, order=keys, **kwds)

gives the desired plot (except for different size). The strip plot can be added on top by calling seaborn.stripplot afterwards:
if multi_panel and groupby is None and len(ys) == 1:
    g = sns.catplot(y=y, data=obs_tidy, kind="violin", col=x, col_order=keys, sharey=False, order=keys, **kwds)
    if stripplot:
        sns.stripplot(y=y, data=obs_tidy, jitter=jitter, color="black")

At the moment, I am just unsure how to plot to each subplot of the new g. It might be possible to loop through the plot grid so not to add everything on top of the last plot.

Besides the not yet solved strip plot problem, would that be a valid alternative?

There is not particular reason to use FacetGrid. If catplot does the job then it is ok.

The following code produces the desired plot (up to permutation of the subplots and the image size) in case of only one row of subplots:
if multi_panel and groupby is None and len(ys) == 1:
    # This is a quick and dirty way for adapting scales across several
    # keys if groupby is None.
    y = ys[0]

    g = sns.catplot(y=y, data=obs_tidy, kind="violin", col=x, col_order=keys, sharey=False, order=keys, **kwds)

    if stripplot:
        grouped_df = obs_tidy.groupby(x)
        for ax_id, key in zip(range(g.axes.shape[1]), keys):
            sns.stripplot(y=y, data=grouped_df.get_group(key), jitter=jitter, size=size, color="black", ax=g.axes[0, ax_id], **kwds)

Seems a bit hacky to me. What do you think @fidelram?

@WeilerP probably there is a more direct way to overlay the stripplot but I don't think that it makes any big difference.
Can you make a PR to see how the tests work?


[MCS-DE-GO] KeyVault Integration for Bot Runtime

All keys and secrets have to be stored in a secure way leveraging Azure KeyVault as a customer requirement. Secret information is not allowed to be stored in unencrypted configuration files.
This is true for at lease all FSI customers in Europe and most other Industries.
KeyVault Integration should be enabled via "appsettings" as boolean plus configuration of the KeyVault URL for the KeyVault to be used.
Add KeyVault as an ICONFIGURATION source.
To enable multiple Bots with the same KeyVault default settingsnames should be prefixed by the botname.
@joaollq @sgellock @benbrown

Adding info from project. We use the AppSettings on the Bot WebApps now and put these into the KeyVault. This is all Azure Services configuration so should be okay. You just need to make sure that the publishing scripts do now store secrets within the appsettings.files on the WebApps themselves. We ensure this by doing the deployment via Azure DevOps Pipelines.
I believe this topic can be solved without major changes mainly by providing proper documentation on how to to is. We can provide input, if needed.

@benbrown @cwhitten Any update on potential timeline for this?

@benbrown Any update on potential timeline for this?

@goergenj can you tell how you implemented azure devops pipelines with bot composer

@rahul-8437008949 What does your question relate to? The KeyVault-integration? We created special Publishing Plug-Ins in Composer which run git commands to push the changes to DevOps and start the Build/Release pipelines. Also triggering pull-requests with approval gates for publishing into productive environments.
The pipelines themselves take settings from the DevOps Libraries at the time of deployment to create the proper appsettings.

@goergenj i also tried deploying code via ado pipelines.The bot gets deployed on azure but bot channel registration doesn't get linked. I thought this is a problem of composer but as you have implemented ado pipeline implementation can you help me out please. For key vault i was trying to use managed identity somehow.

@rahul-8437008949 We are not deploying the Bots Azure Services. They are pre-setup via Terraform. We populate only the Bot code, LUIS, QnaMaker, etc.
If you want to add the Azure Services deployment to the pipeline, I recommend to use an ARM-template which is parameterized so you can run this. I would separate this from the code publishing however, to avoid re-deploying the Azure Services each time.

@goergenj Can you please share a few details regarding how you integrated key vault with ado please.
Thanks.

@rahul-8437008949 Please send me an e-mail to<EMAIL_ADDRESS>Not sure what you need.

Any update on this?


KAFKA-3842: consolidate utility methods to TestUtils, added StreamsTe…

…stUtils, added method for pausing tests to TestUtils
Changes made:

Added utility method for creating consumer configs.
Added methods for creating producer, consumer configs with default values for de/serializers.
Pulled out method for waiting for test state to TestUtils (not using Thread.sleep).
Added utility class for creating streams configs and methods providing default de/serializers.

@guozhangwang - please review when you have time.
Thanks!

A couple general comments:

I think we can just merge the functions in StreamsTestUtils to TestUtils to make it centralized for all unit test util functions.

There are some functions like waitUntilMinKeyValueRecordsReceived in IntegrationTestUtils that can just leverage the newly added waitCondition call.

I think we can just merge the functions in StreamsTestUtils to TestUtils to make it centralized for all unit test util functions.
@guozhangwang

I had originally wanted to do just that.  But the StreamsConfig class is not visible in TestUtils, due to the dependency structure of the streams and clients modules.  I agree it would be nice to centralize into TestUtils. I thought about moving StreamsConfigs into the clients module, but I wasn't sure if that made sense.  Any recommendations?

Ah I see. Yeah package dependency wise we would like kafka-streams to depend on kafka-clients, but not vice versa. In this case, let's keep the StreamsTestUtils for those util functions that need to depend on kafka-streams but leave others to TestUtils.
And regarding 3) above I think we can still make the merge as it does not have any streams package dependency.

@bbejeck I think Thread.sleep is fine here. Just make sure that we will Thread.sleep(Math.min(100L, time-remaining)) in case the remained time is less than the hard-coded backoff value.

@guozhangwang
Changes made per review comments.

LGTM. Thanks @bbejeck ! It'll be great to update the unit tests using the helper functions in TestUtils in kafka-clients, and remove the corresponding duplicate helper functions in kafka-core's TestUtils. Could you file a JIRA?

Merged to trunk.

Keep in mind that a number of kafka-core's TestUtils functions use lambdas so they can't really be replaced.

Also, seems like I was late adding my comments. They can be addressed in a follow-up PR once there is agreement.

Also, seems like I was late adding my comments. They can be addressed in a follow-up PR once there is agreement.

@ijuma   cc: @guozhangwang
Thanks for the additional feedback, should I address these issues in a new Jira ticket?

@bbejeck you can just create a follow-up PR without new JIRA ticket, for example: https://github.com/apache/kafka/pull/1513

@bbejeck Also, feel free to continue the discussion in this PR so that we reach consensus before you make the changes. Most of my comments had questions attached to them. :)

Keep in mind that a number of kafka-core's TestUtils functions use lambdas so they can't really be replaced.

Ack.


why cant i register a embedded function as for shared worker?

why cant i register a embedded function as for shared worker?

What's an embedded function?

A data: URL containing a script?

yes exactly ... as i can do with shared worker :) it would be easy to write
all in a page ... not putting js file in the path of html pages
2016-05-23 18:57 GMT+02:00 Ben Kelly<EMAIL_ADDRESS>
A data: URL containing a script?
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/slightlyoff/ServiceWorker/issues/902#issuecomment-221030729

If you registered a bad data: URL script, how would the browser check for updates to get a fixed one installed?

If you register a bad url , you cam remove manually or programatically. For updates you can verify a new script using md5 of blob function for example. It is trial.

It seems like this would enable persistent XSS attacks.  Being able to execute code once in the origin allows you to control everything that happens in the future on the origin.  In contrast, by requiring a URL, you need to be able to host code on the origin which is a significantly higher bar and requires some intent on the part of the origin.  This is especially relevant in a world where many sites may load third-party JS like ad scripts or analytics.

Yeah this is exactly why we made this restriction.
On Sun, 29 May 2016, 10:57 Andrew Sutherland<EMAIL_ADDRESS>wrote:

It seems like this would enable persistent XSS attacks. Being able to
execute code once in the origin allows you to control everything that
happens in the future on the origin. In contrast, by requiring a URL, you
need to be able to host code on the origin which is a significantly higher
bar and requires some intent on the part of the origin. This is especially
relevant in a world where many sites may load third-party JS like ad
scripts or analytics.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/slightlyoff/ServiceWorker/issues/902#issuecomment-222373642,
or mute the thread
https://github.com/notifications/unsubscribe/AAFtmnjJIyQPOFRiac8W-y_Xte5hFG91ks5qGdNqgaJpZM4Ikpba
.

If you are able to execute a XSS attack in a domain , you can execute code non only inside the worker. It seams a tautology. The body is a function , if you can modify a function you can modify also the remaining scripts on the page.

The point is that Service Workers are:

Cached.
Able to modify all future network requests issued by the origin in question, including: future JS loads, stripping requests by the site to register its own service worker, or just redirecting all traffic to the attacker's own site, etc.

It is indeed a problem in the first case if an attacker is able to execute JS in an origin they should not be able to.  But what you're requesting allows the attack to perpetuate itself after the original intrusion mechanism may have been fixed.  In my previous comment I used the term XSS as a shorthand for cases that are not strictly XSS; the attacker could also compromise JS hosted on third-party domains.
It's a reality of the web and net that:

Sites will have security flaws that allow XSS attacks, despite new technologies like CSP that make it easier to prevent them.
Sites will get hacked.
Dangerous functionality implemented in standards for the benefit of a few reduces the security of all.
Complicating standards and thereby implementations to make potentially dangerous things safer for niche use-cases also reduces the security of all.

In any event, it's not clear what your actual use-case is.  What is it you are trying to do that needs service worker functionality but cannot host a single JS file?  Your use-case may actually want something more "app" like as offered by browser extensions/add-ons.

The solutions are generated by men, not by HTML. If you introduce intelligent FORM field purging , you solve really the problem evoiding to introduce a thousands to tricks that dont solve the problem, but just limiting.
Service worker is with HTM5, you introduce also what i told above and all solved. It is useless if you limiting service worker if the remaining of page is attacked. It is just a tautology. You have to solve the cause not the conseguences.
I know that service worker is a critical point, but all is based if xss is prevented or not. If it is not prevented a hacker will found a trick for evoiding your tricks anyway.


Can you hook it into an existing Laravel installation

Laravel Version: 6
PHP Version: 7.3

Description:
Can you hook this package into and existing Laravel installation  without interfering with existing users table?
Steps To Reproduce:

Yes this package will install into your Laravel installations without impacting the users table, or any other table for that matter.

Yes this package will install into your Laravel installations without impacting the users table, or any other table for that matter.


Add -Name parameter to Publish-PSResource

PR Summary
Complete the -Name parameter for Publish-PSResource.
PR Context
As per the v2 documentation:

Specifies the name of the module that you want to publish. Publish-Module searches for the specified module name in $Env:PSModulePath.
When you specify a module by name, Publish-Module publishes the first module that would be found by running Get-Module -ListAvailable . If you specify a minimum version of a module to publish, Publish-Module publishes the first module with a version that is greater than or equal to the minimum version that you have specified.

Parameter set names were added to differentiate between the 'NameParameterSet' (using the -Name param) and the 'PathParameterSet' (using the -Path param).
Resolves #469
PR Checklist

[x] PR has a meaningful title

Use the present tense and imperative mood when describing your changes

[x] Summarized changes
[x] Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
[x] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.

Breaking changes

[x] None
OR
[ ] Documentation needed

[ ] Issue filed:

User-facing changes

[x] Not Applicable
OR
[ ] Documentation needed

[ ] Issue filed:

Testing - New and feature

[ ] N/A or can only be tested interactively
OR
[x] Make sure you've added a new test if existing tests do not effectively test the code changed

Tooling

[x] I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
OR
[ ] I have considered the user experience from a tooling perspective and enumerated concerns in the summary.

Closing this PR for the reasons stated here.


Subitems in navigation don't save

When I add a navigation item it saves properly. However, if I drag it so it is a subitem, it doesn't save and disappears on refresh.

I figured this out. It was a javascript conflict with something I was loading for the front end that interfered with the AJAX request.


Fix "CyclomaticComplexity" issue in src/Edde/Common/Node/NodeUtils.php

The method toNode() has a Cyclomatic Complexity of 13. The configured cyclomatic complexity threshold is 10.
https://codeclimate.com/github/edde-framework/edde-framework/src/Edde/Common/Node/NodeUtils.php#issue_591ca77a33f85200010003aa

Closing as "who cares".


Do not load Facebook or Google handlers on chinaInfra

Coverage increased (+0.01%) to 63.58% when pulling 3f3606851cd2c4b3c0a4e865ff24f796496f3bc6 on no-fb-loading-china into 0da52f6ac6c77a2c169213585a1ba3c8e0c7fcaa on master.


Merging validators and validation as well as asyncValidators and asyncValidation

It would be better to have just validators and asyncValidator like in angular-formly

can you explain more on this ?

Instead of having validation: Validators.required, I can have validators: { validation: Validators.required}
Benefit: We now have validation property that can be used similar to angular-formly
http://docs.angular-formly.com/docs/field-configuration-object
We should always try to bridge the gap between angular 1 and 2 field configuration as much as possible

Closing this as #165 will take care of this


[5.2] Use getArrayableItems for Collection::only()

This tripped me up a few times as I expected I could pass a collection to only(). It's a minor improvement for a bit more consistency, happy to refactor if not up to project standards.

This is a breaking change is it not?

I wouldn't really recommend using only this way.


Add function to extract all table names from a query string

For example:
imagine below sample data (4 rows) -->
select query from system.runtime.queries gives:
select abc,xyz from schema4.tableB join schema3.tableA on 1=1
with x as (select * from catalogG.schema7.tableY union all catalogR.schema9.tableA) select * from x where j not in (select id from tableH) and not exists (select 1 from catalogR.schema9.tableA where d='a')
select count(1) from tableU
select 1

select get_table_names(query) from system.runtime.queries would give:
expected result (4 rows ie an array showing all tables involved in each query):
['schema4.tableB','schema3.tableA']
['catalogG.schema7.tableY','catalogR.schema9.tableA','tableH']
['tableU']
[]

ie new function get_table_names() would parse all table names from given sql
use case is to get metrics on table usage. ie how many times each table was queried

What is the use case here? Extracting table is not that simple as it requires semantic analysis to happen and also it requires a session configuration that was present during the query execution.
For reference, all referenced tables in the query are mentioned in the query event. Maybe using eventlistener is a better fit for you?

I want to show a bar chart in Superset with count of how many times each table was queried

So using eventlistener sounds like the way to go. You capture tables from event then you can store them in relation database (postgres?) and you can query that in the way you like.
Let me close this issue.


Sema: Fix crash-on-invalid with incorrect use of variadic generic type

Applying generic arguments to a variadic generic type uses the PackMatcher to build a mapping between generic parameters and arguments.
The PackMatcher is symmetric, so there was an unexpected failure mode that wasn't handled: if the variadic generic type had some non-pack parameters, but the argument list was a single pack expansion type, the match would succeed, grouping all of the generic parameters into a single match.
This is non-sensical, so we need to explicitly check for this case and diagnose it. This requires a new diagnostic, since otherwise the general diagnostic we emit for variadic generic type mismatches doesn't make sense, since it's complaining about there being too few generic parameters.
Fixes rdar://116713961 / https://github.com/apple/swift/issues/69012.

@swift-ci Please smoke test


Run no serving monitoring and latest dep updates

Fixes #

Proposed Changes

as per title

Release Note

Docs

/close


Fix binutils install doc on macOS

if test -n "libiberty"; then \
                /bin/sh ./../mkinstalldirs /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`; \
                /usr/bin/install -c -m 644 ./libiberty.a /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.an; \
                ( cd /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory` ; chmod 644 ./libiberty.an ;ranlib  ./libiberty.an ); \
                mv -f /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.an /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.a; \
                case "libiberty" in \
                  /*)    thd=libiberty;; \
                  *)     thd=/usr/local/opt/binutils/include/libiberty;; \
                esac; \
                /bin/sh ./../mkinstalldirs ${thd}; \
                for h in ./../include/ansidecl.h ./../include/demangle.h ./../include/dyn-string.h ./../include/fibheap.h ./../include/floatformat.h ./../include/hashtab.h ./../include/libiberty.h ./../include/objalloc.h ./../include/partition.h ./../include/safe-ctype.h ./../include/sort.h ./../include/splay-tree.h ./../include/timeval-utils.h; do \
                  /usr/bin/install -c -m 644 $h ${thd}; \
                done; \
        fi
clang: error: unsupported option '-print-multi-os-directory'

Thanks for your suggestions. However, I don't think this is an issue we need to consider, because we don't really need -print-multi-os-directory , we just need compile this library on our own platform without considering other OS environments.
This error message will not affect other installation steps, if you go to the /usr/local/opt/binutils/lib, you'll see all libs you need and you can compile BMF succesfully!

if test -n "libiberty"; then \
                /bin/sh ./../mkinstalldirs /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`; \
                /usr/bin/install -c -m 644 ./libiberty.a /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.an; \
                ( cd /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory` ; chmod 644 ./libiberty.an ;ranlib  ./libiberty.an ); \
                mv -f /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.an /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.a; \
                case "libiberty" in \
                  /*)    thd=libiberty;; \
                  *)     thd=/usr/local/opt/binutils/include/libiberty;; \
                esac; \
                /bin/sh ./../mkinstalldirs ${thd}; \
                for h in ./../include/ansidecl.h ./../include/demangle.h ./../include/dyn-string.h ./../include/fibheap.h ./../include/floatformat.h ./../include/hashtab.h ./../include/libiberty.h ./../include/objalloc.h ./../include/partition.h ./../include/safe-ctype.h ./../include/sort.h ./../include/splay-tree.h ./../include/timeval-utils.h; do \
                  /usr/bin/install -c -m 644 $h ${thd}; \
                done; \
        fi
clang: error: unsupported option '-print-multi-os-directory'

Thanks for your suggestions. However, I don't think this is an issue we need to consider, because we don't really need -print-multi-os-directory , we just need compile this library on our own platform without considering other OS environments.
This error message will not affect other installation steps, if you go to the /usr/local/opt/binutils/lib, you'll see all libs you need and you can compile BMF succesfully!

The -print-multi-os-directory option is more relevant in multi-architecture or multi-operating system environments where libraries are organized into different subdirectories based on the target architecture or OS.
For example, on a 64-bit Linux system, running gcc -print-multi-os-directory and will get lib64. This indicates that libraries for the 64-bit architecture are typically found in the lib64 subdirectory of the compiler's library path.
But modern macOS are all on 64-bit system, so i think we don't actually need to consider it?

if test -n "libiberty"; then \
                /bin/sh ./../mkinstalldirs /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`; \
                /usr/bin/install -c -m 644 ./libiberty.a /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.an; \
                ( cd /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory` ; chmod 644 ./libiberty.an ;ranlib  ./libiberty.an ); \
                mv -f /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.an /usr/local/opt/binutils/lib/`gcc -g -O2     -print-multi-os-directory`/./libiberty.a; \
                case "libiberty" in \
                  /*)    thd=libiberty;; \
                  *)     thd=/usr/local/opt/binutils/include/libiberty;; \
                esac; \
                /bin/sh ./../mkinstalldirs ${thd}; \
                for h in ./../include/ansidecl.h ./../include/demangle.h ./../include/dyn-string.h ./../include/fibheap.h ./../include/floatformat.h ./../include/hashtab.h ./../include/libiberty.h ./../include/objalloc.h ./../include/partition.h ./../include/safe-ctype.h ./../include/sort.h ./../include/splay-tree.h ./../include/timeval-utils.h; do \
                  /usr/bin/install -c -m 644 $h ${thd}; \
                done; \
        fi
clang: error: unsupported option '-print-multi-os-directory'

Thanks for your suggestions. However, I don't think this is an issue we need to consider, because we don't really need -print-multi-os-directory , we just need compile this library on our own platform without considering other OS environments.
This error message will not affect other installation steps, if you go to the /usr/local/opt/binutils/lib, you'll see all libs you need and you can compile BMF succesfully!

The -print-multi-os-directory option is more relevant in multi-architecture or multi-operating system environments where libraries are organized into different subdirectories based on the target architecture or OS.
For example, on a 64-bit Linux system, running gcc -print-multi-os-directory and will get lib64. This indicates that libraries for the 64-bit architecture are typically found in the lib64 subdirectory of the compiler's library path.
But modern macOS are all on 64-bit system, so i think we don't actually need to consider it?

Ok, thank you for the clarification. I am able to successfully compile BMF following your instructions, so I agree we don't need to worry about the '-print-multi-os-directory' error when installing binutils. It's worth adding a caveat note in the docs that an error can be expected from the sudo make install step to avoid future confusion.


workflow yaml file

It seems like default .yml file under workflow hasn't updated yet. I've been trying both azure/webapps-deploy@v2 and azure/webapps-deploy@v3 as deploy action, but I get the same error Node.js 16 actions are deprecated. (v4 doesn't seem to exist) When would this get fixed?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 298b87f4-b6eb-5642-7ab5-cd6df709390d
Version Independent ID: 0518cce4-7677-6d63-979d-e0a9e532efc0
Content: Quickstart: Create a Node.js web app - Azure App Service
Content Source: articles/app-service/quickstart-nodejs.md
Service: app-service
GitHub Login: @msangapu-msft
Microsoft Alias: msangapu

@peter5000
Thanks for your feedback! We will investigate and update as appropriate.

Peter, Your feedback has been shared with the content owner for further review. Thanks for bringing this to our attention.

It seems like default .yml file under workflow hasn't updated yet. I've been trying both azure/webapps-deploy@v2 and azure/webapps-deploy@v3 as deploy action, but I get the same error Node.js 16 actions are deprecated. (v4 doesn't seem to exist) When would this get fixed?

Are you still having this issue? @peter5000

Haven't heard back. #please-close


Update contributing

CONTRIBUTING is created in #18 but is empty

Updated to a minimal version but still needs works


Error everytime I try to connect to my Comma3X

I a,m currently running SunnyPilot if that matters but everytime I try to connect to the Comma using OPTK, it gives me the following pop-up:

The logs show the following:
2024-08-25 00:42:08.252 -04:00 [INF] Connected to Comma3 device at <IP_ADDRESS> on port 8022
2024-08-25 00:42:09.022 -04:00 [ERR] Unhandled Exception
System.ArgumentOutOfRangeException: value ('0') must be less than '0'. (Parameter 'value')
Actual value was 0.
  at System.ArgumentOutOfRangeException.ThrowGreaterEqual[T](T value, T other, String paramName)
  at System.ArgumentOutOfRangeException.ThrowIfGreaterThanOrEqual[T](T value, T other, String paramName)
  at System.Windows.Forms.ComboBox.set_SelectedIndex(Int32 value)
  at OpenpilotToolkit.OpenpilotToolkitForm.<>c__DisplayClass20_1.<<ScanDevices>b__2>d.MoveNext() in C:\Users\l-bre\source\repos\OpenpilotToolkit\OpenpilotToolkit\OpenpilotToolkitForm.cs:line 414
--- End of stack trace from previous location ---
  at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_0(Object state)
  at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(Object obj)
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
  at System.Windows.Forms.Control.InvokeMarshaledCallbacks()

Any ideas?

Same issue here

Same for me.

I had this issue a couple of times. I copied the original zip file contents over the working app directory to fix it.


(frontend)[Age 1270]: Raw data missing data in overview

Closes AGE-1270

backend tests failing unrelated to the issue


New commit to fix PEP8 formatting in issue 6 and to merge with new ch…

…anges in Fuel. @vdumoulin

https://scrutinizer-ci.com/g/mila-udem/fuel/inspections/16765e0a-d74d-4630-9c77-ce182423c4ac

Currently the tests fail because the CIFAR100 dataset isn't downloaded when Travis builds.
Add cifar100 to this line, and we should be okay.

@vdumoulin I think there is a merge mistake. I am looking into it...

I think it may just be a punctual failure. I restarted the tests, we'll see what happens.

Yes, it must be, because I just ran:
fuel-download cifar100
fuel-convert cifar100
They both worked fine...

The tests now pass. Go figure...
The only thing left to do now is to make sure we maintain test coverage. There are two things to test: the download script and the conversion script.
This involves creating tests like the cifar10 downloader test and the cifar10 converter test.
You can pretty much copy-paste those tests and adapt them for CIFAR100.

Looking back at the converter test, it's more involved than adapting the CIFAR10 test. Let's leave the coverage as it is and make the converter test as part of another PR.
Once the tests pass, I'll merge.


ref(refName) does not return a wrapper

As docs said, calling ref(refName) should return a ReactWrapper, but it doesn't after enzyme 3.

There's already a PR fix this https://github.com/airbnb/enzyme/pull/1242


remove commented Collectors\Cache at Config\App

Not accepting this one since that's a placeholder to remind me (or anyone else) that it still needs to be developed.


Indent

Hi
I am having an issue with indentation when including content.
The image below is the output destination for compiling:

The image below is the partial that I am compiling:

This image shows the output of what I am receiving. Is this something that I can fix with options?

Can you please help or point me in the right direction on how to fix this issue.
Thanks in advance

maybe, you can do like this.
index.html
<html>
@@include('head.html', {
    "title": "page title"
})
</html>


Compability with DisguiseCraft

It would be nice if ClearLagg didn't remove the mob you're disguised as. Not sure if this is possible though.

It should be posible through interplugin communication(quite difficult), it could also be useful to let the mob spawn when the mob limit is reached. But I don't know exactly how, if someone can put some light to it, it would help.
As for now, don't use killmobs


Added 2 new sources in problem-sets-competitive-programming.md

Added 2 new sites for competitive programming  problem sets.
What does this PR do?
Add resource(s)
For resources
Description
The problem sets of C2 ladders and Atcoder Problems are added.
Why is this valuable (or not)?
Its valuable as its recommended by Grandmasters on codeforces.
How do we know it's really free?
These are wesites which just list problems.
For book lists, is it a book? For course lists, is it a course? etc.
These are problem sets.
Checklist:

[x] Read our contributing guidelines.
[x] Search for duplicates.
[x] Include author(s) and platform where appropriate.
[x] Put lists in alphabetical order, correct spacing.
[x] Add needed indications (PDF, access notes, under construction).
[x] Used an informative name for this pull request.

Follow-up

Check the status of GitHub Actions and resolve any reported warnings!

@eshellman kindly check this PR and merge it. Thank you

@eshellman Kindly Review this PR and Merge it.

These don't seem like "Problem Sets" to me. Care to explain? Remember, the goal of the lists is to help people find things.

These don't seem like "Problem Sets" to me. Care to explain? Remember, the goal of the lists is to help people find things.

The C2 ladders is similar to a2oj ladders already existing in the problem sets file. These are a set of problems chosen from codeforces to help people.
The atcoder problems tries to achieve a similar goal for atcoder site by listing all problems to users thaf can be sorted according to user.

Maybe we should create a category for ladders

Maybe we should create a category for ladders

Hmm...your call. Many don't exist so it will be short one.

Maybe we should create a category for ladders

Should I do this and submit a PR?

renaming the file can create conflicts. maybe after nov 15 or so.
yes, please create a new subcategory. Start with the links from this PR. When that gets merged, create another to move links from the CTF category.

renaming the file can create conflicts. maybe after nov 15 or so.
yes, please create a new subcategory. Start with the links from this PR. When that gets merged, create another to move links from the CTF category.

@eshellman Done. I have created Ladders subcategory. This category will now serve as collections of ladders (which are public collection of good problems hosted on other sites that host problem sets). I have added links from this PR add a few more links that were wrongly kept in other categories.


[com.google.fonts/check/varfont_instance_names] Add ExtraBlack 1000 weight support

The Cairo PR needed a ExtraBlack 1000 weight added, both Dave and Marc approved this, and it looks like it has been added to the wip spec.
In the Cairo PR, the latest gf-bot report using Fontbakery version: 0.7.21 gives a fail for the check com.google.fonts/check/varfont_instance_names:
🔥 FAIL: Check variable font instances have correct names
com.google.fonts/check/varfont_instance_names

🔥 FAIL Instance name "ExtraBlack" is incorrect. It should be "Black" [code: bad-name]

🔥 FAIL Check has either failed or produced a warning. See our wip spec for further info https://gist.github.com/m4rc1e/8f4c4498519e8a36cd54e16a004275cb [code: bad-instance-names]

It seems like this check needs to be updated to support a ExtraBlack 1000 weight.

I 'm working on this one, thanks for reporting, @eliheuer

Thanks! If you haven't seen it yet, https://github.com/googlefonts/fontbakery/issues/2804 is a closely related issue.

yeah!

Thanks Felipe!


Support s2i-core building and testing on Fedora 41

This pull request supports building s2i-core based on Fedora-41.
Some packages, like perl-5.40 are available only on Fedora 41.

@frenzymadness PTAL thanks.

[test]

This pull request blocks https://github.com/sclorg/s2i-perl-container/pull/315

I have to analyse logs :(

It looks like it failing on script cgroup-limits:
$ /usr/bin/cgroup-limits
bash: /usr/bin/cgroup-limits: cannot execute: required file not found

$ ls -la /usr/libexec/platform-python
lrwxrwxrwx. 1 root root 16 Aug 16 13:19 /usr/libexec/platform-python -> /usr/bin/python3
$ ls -la /usr/bin/python3
ls: cannot access '/usr/bin/python3': No such file or directory

[test]

It looks like it failing on script cgroup-limits:
$ /usr/bin/cgroup-limits
bash: /usr/bin/cgroup-limits: cannot execute: required file not found

$ ls -la /usr/libexec/platform-python
lrwxrwxrwx. 1 root root 16 Aug 16 13:19 /usr/libexec/platform-python -> /usr/bin/python3
$ ls -la /usr/bin/python3
ls: cannot access '/usr/bin/python3': No such file or directory

This is fixed by https://github.com/sclorg/s2i-base-container/pull/314/commits/8eb61947648d429564c10c91cb5e167623eb29b8

[test]

[test]

[test]

One more think I realized when investigating differences between container images: the latest tag on quay.io still points to 39 instead of 40, see https://quay.io/repository/fedora/s2i-core?tab=tags

I don't see Fedora 41 available on quay.io. The action did not run yesterday after the merge nor today in the morning.


Feature request: Read some parameters from environmental variables

Feature description
Use case
Is there a specific goal that this would help you accomplish, or can you provide any other context about how you would like to use this feature?
I've developed a stack of Python packages one of which is a high-level API to atiohttp-client-cache for for some specific purposes called AsyncRetriever. It use this in all my other packages for making web calls. Currently, I have two arguments related to caching in all the functions that make web calls. So, to make things much easier, I am to replace those arguments by reading three caching-related parameters using environmental variables: expire_after, disable, and cache_name. This makes things much easier for users to set this globally rather than making the changes for each function.
I am thinking maybe it would be a nice feature to implement in atiohttp-client-cache.
Workarounds
Is there an existing workaround to accomplish this?
Plan to implement
Are you interested in submitting a PR to implement this?
I can do do if there's interest.

I also like using environment variables for config, but generally I think that belongs more at the application or framework level, not in a library. There are so many different ways to provide config (envars, JSON/YAML/INI config files, cloud provider-specific config services, etc.), and the best option really depends on the application.
It's easy enough to set this up yourself, though. You can put something like this in, say, my_app/config.py:
import os
from contextlib import asynccontextmanager
from aiohttp import ClientSession
from aiohttp_client_cache import CachedSession, SQLiteBackend

# The cache backend object can be defined as a singleton,
# and doesn't require an async context to create
CACHE = SQLiteBackend(
    os.getenv('HYRIVER_CACHE_NAME', 'default_cache_name'),
    expire_after=int(os.getenv('HYRIVER_CACHE_EXPIRE', 360)),
)

@asynccontextmanager
async def get_session():
    cache_disabled = os.getenv('HYRIVER_CACHE_DISABLE', '').lower() == 'true'
    async with (ClientSession() if cache_disabled else CachedSession(cache=CACHE)) as session:
        yield session

Then, in some module where you're making requests:
from my_app.config import get_session

async def foo(url):
    async with get_session() as session:
        return await session.get(url)

Does that help?

Very cool project(s), by the way. My day job at Corteva is data science-adjacent, and involves some geospatial data analysis. We've open-sourced a few pieces of that, like rioxarray and geocube. I don't do any watershed analysis myself, but it looks like a lot of the libraries you use overlap with the ones used here. Looks like fun. Good luck with that!

Yes, it was very helpful. Thanks. That's exactly what I was thinking but more elegant!
Ah, I didn't know that you work at Corteva, that's very cool! I actually use rioxarray in my packages. It would be interesting to know the overlaps and hopefully find some common interests that can benefit the community.


Provide build script for compiling contracts instead of uploading wasm files

Problem
ref #143 , the wasm files are big as well, for example https://github.com/gear-foundation/dapps/tree/master/frontend/apps/escrow/src/assets/wasm, plus the wasm files are changing while the development of contracts, so a compile script before dev makes sense tbh
Solution
provide a script in package.json like compile to compile the target contract and cp the meta.wasm to the expected place on development


Don't override local changes that are newer than hub data

i think i want to bake in something to clsModel to check if the data it's trying to save is older than the current record. so if i save something locally on a device and then report that change back to the hub i want it to preserve the local data until the hub data is newer.
will probably only work with things that have a modified field. so i might need to make sure to add said field to models that i want to be able to update in the web interface that need to report changes back to the hub.

added modified check to clsModel->Save().... in theory it's off by default and i think it hasn't broken anything?

didn't break anything

i added this to the lights syncing.... i don't think i broke light syncing from the hub... next i guess i should do the same for the meal planner.... i also kinda need to make the front end updating data better about not reverting back to the old data right away....

ok so lights weren't syncing... but i think i got it so they will sync now but also won't sync if the data is stale

yeah i think i have this working as much as i'm going to....


Fixed off-by-one in OrderingTable, fixed end-of-chain handling in DMA callback

Didn't realize this was going into the pcsx-redux repo ... 🤦


Update line continuations documentation

Not sure when this changed, but this is valid now -
foo:
	echo {{ \
 "test" \
 }}

This is still invalid -
foo2:
	echo {{ \
"test" \
 }}

error: Expected backtick, identifier, '(', '/', or string, but found dedent
 ——▶ justfile:3:1
  │
3 │ "test" \
  │ ^

LGTM!


Ignore this issue :: False Alarm!!!

False Alarm

Are you sure?
It's right here: https://github.com/stephenmcd/mezzanine/blob/master/mezzanine/project_template/settings.py#L197
I just installed from pypi and it works fine.

Sorry about that, just ignore this issue

No problem :-) Let me know if you find anything else.


update the state for a team after receiving a new command

pair program to connect the parser, validator, and reducer and store the new state into the database.
also, return the new state to slack so we can start playing around with the app.

Solved with pr #38


SoulLink pitch document.

Pitch deck link:
https://www.canva.com/design/DAFlpqRRl5U/3tCFH4UUGTgbDDaexcTjoQ/view?utm_content=DAFlpqRRl5U&utm_campaign=designshare&utm_medium=link&utm_source=publishsharelink


Dont have mobile support?

Is possible this component work well in mobile?

@PH7-Jack it works on IPAD, not tested on small device


[BUG] "Operation" column header is missing in Node page

Describe the bug
"Operation" column header missing in Node page
To Reproduce

Navigate to Node page in longhorn UI

Navigate to Volume page

Navigate to Backup Page

Expected behavior
Operations column header should be available in the Node page.
Environment:

Longhorn version: master
Kubernetes version: 1.17.4
Node OS type and version: RKE DO Linux

Validation: PASSED


mul with a float doesn't seem to to the right thing for binary variables

I'm trying to build a sum based on binary variables and coefficients, but the resulting expression doesn't seem to be correct.
let weights = vec![1.1, 1.2, 1.0, 0.9];
let mut vars = variables!();

let xs = vars.add_vector(variable().binary(), weights.len());

let objective = Expression::sum((0..weights.len()).iter().map(|i| xs[i].mul(weights[i])));

I would expect the xs variables to interpreted as ones or zeroes in the sum, i.e., the objective being the sum of the weights of the variables set to 1. However, it appears that the objective is capped at a value smaller than the sum of the weights.

I need to research this better before I say anything more.


Removing old references to org.opencastproject.db.ddl.generate

Removing all references to org.opencastproject.db.ddl.generation
Fixes https://github.com/opencast/opencast/issues/1125

Your pull request should…

[ ] have a concise title
[ ] close an accompanying issue if one exists
[ ] be against the correct branch (features can only go into develop)
[ ] include migration scripts and documentation, if appropriate
[ ] pass automated testing
[ ] have a clean commit history
[ ] have proper commit messages (title and body) for all commits
[ ] have appropriate tags applied

Thanks for the patch. I'll review this later today.

Looks good. Merged.


[BUG] Auto generated SQL statements don't have backtics around table and column names

OS INFO : Any OS

SQL statements generated by extension don't use backtics, which breaks them when table or column names are reserved words.
Example:

update to 2.3.9 fix.

Works great, thank you!


Unable to run the prediction script

I am using Windows and when I try to run the prediction script, I am getting the error:
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: error while creating mount source path '/c/Users/Prawigya/Desktop/acm/image-quality-assessment-master/src/tests/test_images/42039.jpg': mkdir /c: file exists.
The prediction script used:
./predict
--docker-image nima-cpu
--base-model-name MobileNet
--weights-file /$(pwd)/models/MobileNet/weights_mobilenet_technical_0.11.hdf5
--image-source /$(pwd)/src/tests/test_images

I have tried a lot of things from the internet to get till here, there were major issues I was facing earlier, but now I can't really seem to find a solution to proceed; anyhow if I come across one, I'll let you know. :)


Markdown tip on statusbar panel uses different background color for backtick-marked text

I am using Dark+ theme.

Maybe this is intentional, but I thought I'd check.
VS Code version: Code - Insiders 1.74.0-insider (493a541735b84de36b628750ba716a430669cb7f, 2022-12-02T05:21:11.264Z)
OS version: Windows_NT x64 10.0.22621
Modes:
Sandboxed: Yes

Thanks for creating this issue! We figured it's missing some basic information or in some other way doesn't follow our issue reporting guidelines. Please take the time to review these and update the issue.
Happy Coding!

Do you need more info from me? If so, what?

This is just a screen shot. Repro steps? Yes, status bar items support markdown hovers in case that's the question

Repro steps:

Clone https://github.com/gjsjohnmurray/vscode-extension-samples/tree/show-167917
Open statusbar-sample folder.
Build and run.
In Extension Development Host window use File\New Text File
Enter a few lines and select at least one of them.
Hover over the statusbar panel which the sample adds:

Observer that when color theme is "Dark +" the monospaced phrases in the tip (backtick-marked and #167917) have a different background color compared to the rest of the tip.

https://github.com/gjsjohnmurray/vscode-extension-samples/blob/show-167917/statusbar-sample/src/extension.ts#L38 tells me that markdown rendering is wanted, e.g this isn't an accidental MD rendering. I believe the background style (for inline code) is also wanted but I let @daviddossett confirm

I think it's correct that text within backticks gets a special rendering. Just like it would like this.
I don't however know where these styles are coming from. At a glance, it looks to me like the background should be lighter, not darker, at least in the default dark theme.


Split Web & Mobile screens as they vary too much

We should have a way for Web* to not build into Kt/Swift, and for Mobile* stuff to not build into TS


Revisit resolveRemoteUri vs resolveRemotePath

I recently came across spaghetti-code in windowsMainService around opening/handling remote paths and or URIs. I tried to cleanup and extract the remote handling and the local handling.
Then I noticed that we also distinguish between URI and string paths on startup. We even support remoteAuthority when you have a string, which I do not fully understand how that is possible (can you provide a path as CLI argument and a remote authority to use?)
Anyway, what is fishy is that both methods seem to massage remote URIs but different:
https://github.com/microsoft/vscode/blob/4c0a4179e6e170a2f45526adfff434a5a7e00a07/src/vs/platform/windows/electron-main/windowsMainService.ts#L844-L851
and
https://github.com/microsoft/vscode/blob/4c0a4179e6e170a2f45526adfff434a5a7e00a07/src/vs/platform/windows/electron-main/windowsMainService.ts#L920-L945
I wonder if both should align and share the same logic.

We support the following:
code --remote wsl+Ubuntu /home/jim/projects/c : The path is a path on the remote and because we can not stat it, we need to  'guess' if it is a folder, workspace or file
That's what the second code snippet is about (line 921)
Also supported is
code --folder-uri vscode-remote://wsl+Ubuntu/home/jim/projects/c

We support the following:
code --remote wsl+Ubuntu /home/jim/projects/c : The path is a path on the remote and because we can not stat it, we need to  'guess' if it is a folder, workspace or file
That's what the second code snippet is about (line 921)
Also supported is
code --folder-uri vscode-remote://wsl+Ubuntu/home/jim/projects/c

I did some more polishing of the open the code when I added remoteAuthority as argument for WindowsMainService.open.
There's admittedly lots of spaghetti left, but I think it's in a ok state. No further refactoring planned from my side.


Enable HTTP keep-alive by default for HTTP client

This changeset enables HTTP keep-alive by default for the HTTP client. HTTP keep-alive (reusing persistent connections) offers significant performance improvements when sending many requests to the same host as it avoids recreating the underlying TCP/IP connection and repeating the TLS handshake for secure HTTPS requests.
HTTP keep-alive is now enabled by default. If you do not want to use HTTP keep-alive, you can disable this feature by sending a Connection: close HTTP request header or by using HTTP/1.0 like this:
// not recommended: add `Connection: close` request header to disable HTTP keep-alive
$browser = $browser->withHeader('Connection', 'close');

// not recommended: use HTTP/1.0 to disable HTTP keep-alive
$browser = $browser->withProtocolVersion('1.0');

We've done some careful testing in PR #486 and our sophisticated test suite that covers tests against both well-behaving and misbehaving HTTP servers that might occur in the wild. On top of this, we've done some extensive tests in real-world applications in the past weeks after merging this feature. The test suite confirms this has 100% code coverage and does not otherwise affect our public APIs. If you encounter any issues with this in the wild, please confirm by disabling HTTP keep-alive and report a bug ticket in this case. If you'd like to support this development, consider sponsoring ReactPHP! ❤
Builds on top of #486, #484 and others
Refs #488, #468 and others

:rocket: :rocket: :rocket: :rocket: :rocket: !


Merge from Development to Master

What this PR does / why we need it:
Merge a fix from Development to Master to get ready for m-1 release.
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

Coverage remained the same at 41.659% when pulling 266321d40fbdefb7e1c7ef6dcfb436e6e2bfe207 on development into acbf88e53dad1ba85ac46db7061c2a4ffd2bd722 on master.


[Question] Execption Settings

How can I add a new category in Exceptions Settings?
Visual Stduio 2019 16.9

Can anyone help me?


Update readme copyright years. Fixes #111.

Codecov Report

Merging #113 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #113   +/-   ##
=======================================
  Coverage   93.81%   93.81%
=======================================
  Files           9        9
  Lines          97       97
=======================================
  Hits           91       91
  Misses          6        6

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7411455...ae3f09d. Read the comment docs.

thanks @JonaMata !


Azure support for Snowplow [discussion]

Can someone implement support for running Snowplow on Azure?

+1

I think there are two ways we could implement Azure support:

It should be straightforward to run the planned Kafka+Samza version of Snowplow on Azure boxes. Then we could just implement support for SQL Data Warehouse when that's available
We could create an "Azure idiomatic" version of Snowplow using EventHub

Has anyone successfully used EventHub in the wild? It looks pretty immature compared to Kinesis and Kafka...

Ooh, that sounds promising! How far along is the Kafka+Samza version?
As @alexanderdean had mentioned, the other alternative is to support EventHub. My team has used EventHub successfully in the past pumping stuff in like logs and GPS data. I haven't personally used Kinesis so I can't compare.
If we can use the Kafka+Samza version, a good starting point might be to add support for sinking data to Azure Blob Storage (Azure equivalent to S3). Azure Blobs also have an HDFS interface to which we can run Hive / Pig jobs on top of the blobs.

Hey @jpoon ! EventHub looks interesting but I could only find .NET examples for it; we would need a JVM SDK for it to make it work with Snowplow.
The Kafka+Samza version is still currently on the drawing board - although we do have a branch with Kafka support for our Scala Stream Collector.
One potential architecture for Azure could be like this:
trackers -> Scala Stream Collector -> Kafka -> Kafka-Blob sink -> an Azure-friendly version of our existing Hadoop flow

I guess questions I would have:

Is there an equivalent of Pinterest's Secor but for sinking from Kafka to Azure Blob Storage instead of S3?
Our EmrEtlRunner Ruby (soon to be JRuby) app does a lot of Hadoop orchestration on Elastic MapReduce, using Elasticity. What is the equivalent for programmatically orchestrating a Hadoop job on Azure? We would need a Ruby or JVM SDK here too

As per the architecture you listed, just want to make sure we're on the same page with all our options:
Collectors

Event Hub: Data can be pushed into Event Hub through the REST API or via AMQP. Apache Qpid has a JMS component that we could use to pull data in and out. Also came across some examples here.
Kafka: If we use Kafka, it's not cloud-specific so we could run it on an Azure VM and start partying.

Storage

Azure Blob Storage. I couldn't find an existing solution out there for sinking Kafka/Event Hub to Azure Blob Storage. If we can't find one, I don't foresee this being too difficult to implement. (famous last words? :P)
SQL Data Warehouse. Not entirely sure when public preview will be open. Might be a good goal in the long term?

Analytics/Compute

I haven't worked a whole lot with Hadoop. The Azure version of Hadoop is known as HDInsight which is essentially repackaged Hortonworks Data Platform (HDP). Let me try to find out if there is something simliar to Elasticity.

Hey @jpoon ! Thanks for the summary of options, very helpful. Some comments:

EventHub - the hdinsight/eventhubs-client looks promising. Potentially we could embed that into the Scala Stream Collector, and then Snowplow raw events could be written straight to EventHub
Kafka or EventHub -> Azure Blob Storage. Shame there isn't something already - it feels like this would be a good thing for Microsoft to build to encourage Azure adoption. A bit like Amazon's amazon-kinesis-connectors library. We have written kinesis-s3, somebody needs to write kafka-azureblob or eventhub-azureblob :-)
SQL Data Warehouse - yes, having a horizontally scalable analytics database available is essential for a Snowplow user. How do we get on the private beta? :-)
PostgreSQL - yes, PostgreSQL support is available. It will struggle with any serious event volumes, but it can be a good way of testing things.
HDInsight - let me know when you know about orchestration tools. The other thing we lean on heavily is AWS's S3DistCp, which is an Amazon fork of the regular Hadoop DistCp tool, which makes it easy to move data S3<>HDFS. I wonder if there's an equivalent in HDInsight?

I've done some digging, and it looks like there is a .NET library for automating Hadoop jobs on HDInsight:

https://hadoopsdk.codeplex.com/
https://azure.microsoft.com/en-gb/documentation/articles/hdinsight-submit-hadoop-jobs-programmatically/#submit-mapreduce-jobs-using-hdinsight-net-sdk

Our use case:
I'm interested in trying out Snowplow but I want to do so in Azure because that's where we have a lot of free credits.
What's the shortest path for me to do this on Azure?
We're a small team and just starting out with tracking user analytics so we want something quick to setup but where we'll own the data and will be easy to scale in the future without a migration.

As per the above, it's not possible currently...

Kafka or EventHub -> Azure Blob Storage. Have you found that a lot of people persist the raw events from Kinesis straight to S3? What is the difference between Amazon's version vs snowplow? In any case, agreed that it would be useful code. I wouldn't mind giving a crack at it once we decide which direction to go -> Kafka vs Event Hub.
SQL Data Warehouse. Let me poke around the company :)
HDInsight. HDInsight has native support for Azure Blob Storage (ie. no need to move it to an explicit HDFS store). Once the raw events are in Blob Storage, we can run MR jobs directly on top of it. As for management libraries, as you pointed out, there's the client .net library. Otherwise, as HDInsight deploys the Hortonworks Hadoop stack there is Ambari and WebHCat out of the box.

I think what @thomaswrenn may have meant is in consideration of the options above, quickest path to supporting Azure. In this case, it might mean adding support for kafka or event hub and then using the persisting that data to PostgreSQL? What do you think?

Yeah, sorry, I severely miswrote. I mean’t to phrase it as what the smallest set of implementations/ports would be to get something running on Azure like @jpoon said.

Hey @jpoon, @thomaswrenn  ! No worries at all. Answers to your questions/comments:

Have you found that a lot of people persist the raw events from Kinesis straight to S3?

Yes - at the moment we don't do loading of Redshift from Kinesis (only from Hadoop), so currently we recommend people sink the raw events to S3 and then run our Hadoop pipeline to load into Redshift as per our latest release blog post
In any case - yes archiving raw events is important to facilitate re-processing later; I explore this in chapter 9 in my book.

What is the difference between Amazon's version vs snowplow?

Amazon's version is more of a toolkit (albeit with a sample application) - kinesis-s3 is a pre-built app which just needs a configuration file. Also, kinesis-s3 writes lzo-index files which are very Hadoop-friendly.

HDInsight. HDInsight has native support for Azure Blob Storage (ie. no need to move it to an explicit HDFS store).

Ah okay. EMR has native support for S3 too - but moving to the cluster's HDFS with S3DistCp first makes processing much faster. Is that not the case with HDInsight?

Otherwise, as HDInsight deploys the Hortonworks Hadoop stack there is Ambari and WebHCat out of the box.

Right - and now that the Snowplow Hadoop job runs on Hadoop 2.4.0 / YARN, we could probably submit the job using the YARN Client?

quickest path to supporting Azure ... it might mean adding support for kafka or event hub and then using the persisting that data to PostgreSQL?

My gut says:

Scala Stream Collector with EventHub support via eventhubs-client
EventHub -> Azure Blob Storage sink app
Snowplow CLI app can submit Hadoop job to vanilla YARN cluster (#1792)
We extend our Redshift loading engine to also work with SQL Data Warehouse

This is the minimum number of moving parts which is still very horizontally scalable.
Postgres would be less work but it would be much less impactful - because nobody with serious event volumes can use Postgres with Snowplow. Also a powerful part of Snowplow is the custom shredding process which loads our event JSONs into dedicated tables in Redshift...

Do you have a recommended blog post or design document that explains the shredding?

These are probably quite a good place to start:

http://snowplowanalytics.com/blog/2014/07/09/snowplow-0.9.5-released-with-json-validation-shredding/
https://github.com/snowplow/snowplow/wiki/6-Configuring-shredding

HDInsight + Azure Blob Storage

Best practice is to have the data in Blob Storage. Azure did a massive amount of network upgrades a few years ago such that for MR jobs, perf characteristics for accessing data in Azure Blobs was near identical to the cluster's HDFS.

HDInsight + YARN

Totally :+1:. It's probably best to avoid any specific client libraries where we can such that anything we do here can also apply everywhere else.

PostgreSQL

Fair enough.

To throw another option into the ring, an alternative to building an explicit app for EventHub->Azure Blob Storage sink is to leverage another SaaS offering from Azure: Stream Analytics . We can use it to (1) read from event hub, (2) do some sort of processing, if any, (3) push it to azure blob storage.
With that in mind, we could build snowplow support for:
event hub -> stream analytics -> azure blob storage
                              -> sql data warehouse

Hmm - streaming analytics doesn't look like you can execute arbitrary code. We need a runtime (whether Hadoop, Samza or something else) where we can embed all our enrichment and validation logic :-)

Azure Data Factory looks like the 'azure way' for managed ETL's, could this be a piece of the puzzle? https://azure.microsoft.com/en-us/services/data-factory/

Potentially @philbritton ! But these sorts of managed cloud ETL services tend to be too opinionated to make a port of Snowplow straightforward...

I would really like Snowplow support in Azure.
Now Storm and Spark Streaming is available as part of Azure HDInsights. Also there is MPP version of Postgres (XtremeData) available in Azure Marketplace. So you could technically use Redshift loading engine with XtremeData without huge changes.
So probably this could be a viable alternative.
event hub -> storm/spark streaming -> azure blob storage
-> Xtremedata (Postgres MPP)
-> azure SQL Datawarehouse
Can't wait to see Snowplow on Azure..

Hey @rgknp - we would love to see Snowplow on Azure too! How do we make this happen? Storm/Spark Streaming is a nice idea but we always want support for our standard batch flow so that we can enable users to perform full-archive re-processing (e.g. when we introduce new enrichments or update the enriched event format).
So we need to support at a minimum:
collector -> event hub -> azure blob storage -> hadoop -> azure sql data warehouse

With this in mind - has Azure made any progress in the last 12 months with providing a standard component for sinking an EventHub into Azure Blob Storage (think Secor but for Azure)? We are only a small team and don't have the bandwidth to build these kinds of generic components for Azure at this time...

Yes, Azure now has EventProcessorHost component for receiving events from Event Hub. Below URL has complete implementation including code.
[(https://azure.microsoft.com/en-us/documentation/articles/event-hubs-csharp-ephcs-getstarted/)]

Hey @rgknp - the EventProcessorHost looks pretty low-level. AWS provide the kinesis-connectors project to make things like Kinesis->S3 straightforward. What's the Azure equivalent?

@alexanderdean Not sure it fits your scenario but we're using stream analytics to stream data from event hubs to blob/table storage.

Thanks @markusahlstran - that's helpful to know. I read somewhere that Stream Analytics has to deserialize the individual records before working with them - isn't that rather inefficient if we just want to write them to Blob Storage?

@alexanderdean what is the performance you are looking for? In Stream Analytics you could do "select * from  and output it to blob storage. Each streaming unit can give throughput of 1 MB/sec. You can add streaming units to scale up to 1GB/Sec.

That sounds good @rgknp - does it work with Thrift? How are records written to Blob Storage - presumably N incoming records are written to each file - what is the file format for storing N records?

@alexanderdean Stream Analytics supports Avro, CSV and JSON file formats. I am not aware of Thrift support. But you may want to look at Bijection APIs https://github.com/twitter/bijection for Thrift -> Avro conversion if needed.
Records will be written to blobs which can be in user defined containers. You could define path prefix, date and time format if you want data to be split into separate blobs. Each blob can hold max 1TB of data so it is a good idea to split records based on some kind of distribution. Splitting blobs based on date and time is most popular. More details can be found here https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-define-outputs/

@alexanderdean I found kinesis-connector like libs for Azure. These are called data-pipeline libs. Written and supported by Microsoft.
https://github.com/mspnp/data-pipeline
https://github.com/mspnp/data-pipeline-storm

Great find @rgknp !

I am glad that worked out. Can't wait to have Snowplow on Azure.
@alexanderdean do you have a timeframe for this?

No timeframe, sorry - getting Kafka support for Snowplow is our first priority.

@alexanderdean Do you have/plan to have Azure in your roadmap?

Hi @rgknp ! It's not yet on the roadmap. We are currently still at the point of trying to identify internal sponsors at Azure for our port. But did have a great meeting @ MS last week in London where we went through Azure SQL DW - very cool tech...

Sounds good!

Discussion migrated to http://discourse.snowplowanalytics.com/t/porting-snowplow-to-microsoft-azure/1178


Open XML SDK v3: struct IdPartPair

Describe the bug
It would be nice to mention all v3-breaking changes in the release notes clearly.
Screenshots
Bug with the fix.

To Reproduce
IdPartPait has been converted from class to struct in v3 (most likely for performance reasons).
This change silently broke code that worked fine before and checked the value with null.
After the upgrade, the code compile without errors, but some branches became unreachable and app/library no longer works as expected.
Observed behavior
No release notes yet ;)
Expected behavior
Mention such cases inside the Breaking Changes section in released notes.
Additional context
I believe that it will be helpful for apps and library authors who decide to migrate to v3.
Found during migrating my fork of PowerTools to SDK v3

Just found that it was mentioned in the release notes for the beta1.
But I personally did not realise from this line how it would affect my code.

Hey @sergey-tihon,
Perhaps you're fine with this now but I thought I'd mention that I see that the compiler does at least inform you that this will always be true:

Yes compiler informs with a warning, that it is truly hard to catch If you do not know that something will break.
I think that would be nice to mention in the release notes (breaking changes section) what warnings should be treated as errors during the migration to v3.

Here's the docs for it: https://github.com/OfficeDev/open-xml-docs/blob/live/docs/migration/migrate-v2-to-v3.md.
I thought I had put a before/after recommendation on how to change this - but apparently it didn't make it there. Happy to take contribution there :)

we consider this issue addressed. Please see twsouthwick's comments. Closing.


refactor

What does this PR do?

Refactoring as per comment on https://github.com/huggingface/accelerate/pull/667#discussion_r992310418

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


why on the earth ui is served from internet?!

Hello, dear author!
Your program is definitely neat and perform better than competitors in some aspects
BUT
why on the earth ui is served from the internet?! https://ui-v000.eqmac.app (which resolves to cloudflare) and here we go, oh my. I can't say a word, how I'm surprised by that move.

Because i can deploy over the air updates and bug fixes without doing a whole native update.
The UI is cached with service workers so works offline as well.  Also the UI is shipped with the app in case user open eqMac for the first time without internet.
So what's your problem with that?

So what's your problem with that?

Literally, my equalizer is tracking me and uploading (unknown portions of) data to servers. :)
At least, you could make that clear to users, IMHO.
Other than that, I do not have problem with it. Keep it up!

How is it unknown if you can see exactly what I'm sending to Google Analytics here:
/ui/src/app/services/analytics.service.ts
How would a non Web based UI stop me from sending any data back to servers?

Why was this labeled with a bug label @aol-nnov ?

These types of questions should be discusses in our Discord server. GitHub is for actual problems, not a place to express any random thoughts.

Why was this labeled with a bug label

Cuz you made some issue templates, obviously! :)
This is a privacy concern, not a feature request.


Sentry\captureException doesn't work.

i have small code run on cli. an php-resque implement.  after initting Sentry, any unhandle exception is sent to sentry.
but the exception that be try catch doesn't. for example:
Sentry\init(['dsn' => 'sentry dns']);

$application->bootstrap();
// add sentry
$hello = new Exception('hello! i am an exception');
Sentry\captureException($hello);

=> nothing happen, i saw the proxy, and didn't see any outbound request. the sentry dashboard still empty.
i try another version without handling exception.
Sentry\init(['dsn' => 'sentry dns']);

$application->bootstrap();
// add sentry
$hello = new Exception('hello! i am an exception');
throw $hello;

it work, have an outbound request to sentry server. and sentry dashboard shown the error.
sentry sdk version: 2.1.0
is that a bug?

after debugging on sentry sdk. i found that: by default, the exception will be sent after shut down php. but my cli, i keep it never shutdown, so, the request be keep in sdk till php shutdown.


docs: Show typing information in live editor

Why not have lightweight VSCode editor in the docs site itself?

typeahead
hover type inspection

Codecov Report

Merging #2236 (30d7d78) into master (fb6f578) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #2236   +/-   ##
=======================================
  Coverage   98.54%   98.54%
=======================================
  Files         114      114
  Lines        1860     1860
  Branches      273      273
=======================================
  Hits         1833     1833
  Misses         10       10
  Partials       17       17

Impacted Files
Coverage Δ

...act-integration/provider/ExternalCacheProvider.tsx
100.00% <ø> (ø)

...e/src/react-integration/provider/CacheProvider.tsx
100.00% <100.00%> (ø)

...core/src/react-integration/provider/CacheStore.tsx
100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Expand target features section of the book

I've written some stuff for the target features section of the book.

I'm closing this PR (temporarily?) and moving the discussion to the tracking issue #132.