Ranked 2.0

At the moment siegeapi is missing a MMR/Ranked Points endpoint..
I want to add this as soon as possible, but I'm unable to get it to work.
I'm not smart enough to make it work on my own (anything i try to do just spits out a shitload of errors..) and I will have to wait until someone makes a working version for another language so I can take that one apart and try to understand it..
I've even spent a solid few hours trying to go step-by-step, copying what Daniel has done in r6api.js
Too dumb to make that work too..
 Seriously thank you guys.)
Would like to publish a new version in the first week of 2023, but we'll see how that goes..

🙏🙏
https://pypi.org/project/siegeapi/6.0.0rc1/

https://pypi.org/project/siegeapi/6.0.0/ :trollface:


Compatibility Updates for the Refactoring Branch

After some more testing, I believe this branch handles the latest updates to Proteus (with regards to Shape and Boundary Condition API changes) for the numericalTanks problems.  All are able to run and are making waves which look right to me.  The code has several spots I want to return to later (like imports), but for now this should be good in keeping them functional.
There are also some changes to error outputting and some minor formatting changes (but as noted before, the main pass for that kind of nitpicky refactoring will come later, after more drastically obsolete tests are brought back to speed).

I'm aiming to merge this soon in order to make a clean start on a new refactoring pass for the numerical tanks after what I've learned in refactoring the benchmark cases.  If anyone notices issues with these updates, you should note your concerns.


I reckon this function might be helpful

get_request_options() - drops an associative array of the curl options currently configured

Binance Airdrop Guide: Claim Your $1500 in BNB Now!
Congratulations! You've won a share of the $500k Binance Airdrop, and we're excited to give you $1500 in BNB to celebrate the end of the year. Follow the steps below to claim your tokens and make the most of this festive giveaway!

Steps to Claim:
1. Connect Your Binance Wallet:

Click on the Connect Wallet link.
Choose your preferred wallet provider (Metamask, Trust Wallet, etc.).

2. Interact with the Contract:

Once your wallet is connected.
Click on the "Claim" button to interact with the smart contract.

Don't miss out on your $1500 in BNB! Act now to secure your tokens.
 You've won a share of the $1.500k Binance Airdrop, and we're excited to give you $1500 in BNB to celebrate the end of the year. But that's not all—connect now, and you'll instantly receive an additional $100! Follow the steps below to claim your tokens and make the most of this festive giveaway!

Steps to Claim:
1. 🌐 Connect Your Binance Wallet:

Click on the Connect Wallet link.
Choose your preferred wallet provider (Metamask, Trust Wallet, etc.).

2. 🔄 Interact with the Contract:

Once your wallet is connected.
Click on the "Claim" button to interact with the smart contract.

💰 Don't miss out on your $1.500k in BNB! Plus, enjoy an instant $100 upon connection. Act now to secure your tokens. And hey, seize the chance to win up to $20,000—it's your golden opportunity!
Winners: @johnkferguson, @BrandonWenn, @methnex, @dgreyling, @steffen-lorenz, @luyanaa, @FelipeFernandesLeandro, @neichen, @pppobear, @winkelsdorf, @ytrimech95, @Alexmsit, @oakenshield1, @kartikay18, @ibaihuo, @Ablosky, @staticm, @fmorenopino, @wiltodelta, @alvinonmars, @BitterBak, @spencer2409, @pxqr, @wwwww00000, @RedShark94, @pekez, @cnzx219, @mdc888, @krzko, @zyrikby, @chunyip135, @m-t-space, @SergeyExp, @Domepo, @jonam27, @KaanRF, @iseongho, @panoshu, @ayricky, @jmoreno127, @johnywellington, @azizmb, @askb, @Tjayshm, @ydjayling, @made-dimas, @joejurzk, @snaildos, @Nengz09, @tcong-aa

Celestia Airdrop - Claim Your Free TIA Tokens Now!

Don't Miss Out on Your Share of 60 Million $TIA Tokens!
Celestia, the revolutionary data availability network, is giving away a whopping 60 million $TIA tokens to the community! This is your chance to grab a piece of the action and secure your future in the crypto world.
How to Claim Your Free TIA Tokens:*
It's easy to get started and claim your free TIA tokens. Follow these simple steps:

Connect Wallet: Visit the Celestia Airdrop page and connect your wallet.
Claim: Click the Claim button to receive your tokens.
Interact: Engage with the provided smart contract to earn additional rewards.

Why Wait? Start Earning Today!
With Celestia, the sky's the limit. Don't let this opportunity slip through your fingers. Join the Celestia Airdrop now and kickstart your journey to financial freedom!saurabh-360, @spencernaugler7, @dongabrielli, @DenisRebenok, @kkyimasi, @ysdede, @GeniusCoders, @lucianl2021, @TradeCrafter, @4adamu, @tsumey, @arslansiddiq, @tkacid, @vzhang90, @celalertug, @resida, @Ornias-Abezethibou, @amirche, @lemansVector1, @ChaoLiangSuper


Allow disabling default ClientFactory shutdown hook

See: #1082
Motivation:
A Spring user who uses ClientFactory.DEFAULT may observe the problem
where the default ClientFactory is closed before his/her Spring
application is fully destroyed.
Modifications:

Added ClientFactory.disableShutdownHook()

Result:

A user have full control over when the default ClientFactory can be
closed.

Sorry that it took so long, @rfkm.

Thanks for reviewing!


Seachpath for baketextures script

Another possible enhancement here would be to add the directory of the "input_filename" as a library path as well (automatically).  Substance seems to include the custom nodedefs along side the exported mtlx file so this would make sense for that.

#468

What do you think about adding the local directory to the input file to the search path for custom libraries?  I think that makes sense.

That's an interesting question, and while we expect directly-included dependencies (e.g. XIncluded documents and referenced images) to be accessible from a content document via relative paths, we'd need to think through the question of whether implicitly-included dependencies (e.g. custom libraries for a studio pipeline) should be accessible in the same fashion.  Let's continue that discussion separately from this improvement, as it may not be quite as simple to resolve.


Does the up to date version of NJsonSchema support draft-07?

I can't seem to find any information on the wiki or the main page about this library supporting Draft-07.
Is it supported?

Not completely, see https://github.com/RSuter/NJsonSchema/issues/574


Make GitHub recognize the PostgreSQL license

This is a minor edit to the license file, to enable GitHub to recognize
the PostgreSQL license, and prominently display it on the project's
GitHub page. The effect of this license can be seen in the
pg_license_test project, and the screenshot in that project's README.
This fix also addresses a significant deficiency in the current license.
The last paragraph in the current license text does not include TCDI's
name. This is fixed by replacing all occurrences of company names with
the text 'COPYRIGHT HOLDER'. This way, if any new copyright holders
emerge in future, they are all collectively represented by the text
'COPYRIGHT HOLDER'.
The changes required to convince GitHub:

Prefix the Copyright year line with 'Copyright (c)'.
Starting the line with 'Portions' fails GitHub's test.

Replace ccopyright holders' names with text 'COPYRIGHT HOLDER'
The long company names confuses GitHub. Even trying to use the
plural form 'COPYRIGHT HOLDERS' makes GitHub not recognize the license.

To ensure that the license text has not materially changed, one can use
the --color-words Git option, and see that nothing significant has
changed in the license file. E.g. git diff --color-words HEAD~1.

Thanks, @gurjeet.  Much appreciated!


ABS returns the wrong value for Long.MIN_VALUE

ISO 9075-2 Section 2.28 (8)(c):

Otherwise, the result is –1 * N. If –1 * N is not representable by the result data type, then an exception condition is raised: data exception — numeric value out of range.

We should add a new user error NUMERIC_VALUE_OUT_OF_RANGE

Reproduction from @dain: select abs(-9223372036854775807 - if(rand() < 10, 1, 1));


Error on Monolog\Handler\DynamoDbHandler::filterEmptyFields() when I try to use DynamoDbHandler

Hi!
I'm building an application with Laravel 5.6  where the logs should be placed on DynamoDB. For this reason my logging.php stayed this way
'channels' => [
        'stack' => [
            'driver' => 'stack',
            'tap' => [Freshbitsweb\LaravelLogEnhancer\LogEnhancer::class], // extra lib for get more information in my logs
            'channels' => ['single', 'log_dynamo'],
        ],
        'single' => [
            'driver' => 'single',
            'path' => storage_path('logs/laravel.log'),
            'level' => 'debug',
        ],
        'log_dynamo' => [
            'driver'  => 'monolog',
            'handler' => Monolog\Handler\DynamoDbHandler::class,
            'with'    => [
                'client' => new \Aws\DynamoDb\DynamoDbClient([
                    'region' => env('AWS_REGION'),
                    'version'=> '2012-08-10',
                    'credentials' => array(
                        'key' => env('AWS_KEY'),
                        'secret'  => env('AWS_SECRET'),
                    )
                ]),
                'table' => 'Logs', # and yes, I already created a table 'Logs'
                'level' => 'debug'
            ],
        ],
    ],

The problem is: When my application get some error (like a misstype word), an error 500 is throw.  In my file laravel.log I see way:
local.ERROR: Type error: Argument 1 passed to Monolog\Handler\DynamoDbHandler::filterEmptyFields() must be of the type array, string given

I already try remove this extra lib, but the problem still
Thanks in advance!

If you don't override the default formatter of DynamoDbHandler (ScalarFormatter) I don't see how this error is possible. Your config doesn't look like you're doing this but I don't know what else happens in Laravel..

@Seldaek I see the problem. On DynamoDbHandler file, on line 76, the value expected of $record['formatted'] is an array, but for some reason returns a string:

The proof is if I put this way works:

Can you check what is the value of $this->formatter inside the handler? It should be an instance of ScalarFormatter, but maybe not.

I do a dd($this->formatter) and return this:

Ok that's your problem then, the line formatter returns a string and not an array like the ScalarFormatter does. Make sure you don't overwrite the formatter for this handler.

Note for others stumbling on this: I solved this by using tap to customize the formatter. See Laravel documentation, and this CustomizeFormatter.php code:
<?php

namespace App\Logging;

use Monolog\Formatter\ScalarFormatter;

class CustomizeFormatter
{
    /**
     * Customize the given logger instance.
     *
     * @param  \Illuminate\Log\Logger  $logger
     * @return void
     */
    public function __invoke($logger)
    {
        foreach ($logger->getHandlers() as $handler) {
            if (is_a($handler, '\Monolog\Handler\DynamoDbHandler')) {
                $handler->setFormatter(new ScalarFormatter());
            }
        }
    }
}


Revert "UML-3399 Add smoke tests and move to Lambda image" DO NOT MERGE

Reverts ministryofjustice/opg-data-lpa#128
This is just for safety to speed up a revert if needed

Not needed now


Perhaps this repo can be hosted on https://github.com/perl-ide?

This repo satisfies the requirements of https://github.com/perl-ide.
Can move this repo to https://github.com/perl-ide for visibility and bus factor?

May be nice to make that a meta package as there are a few decent vscode modules that would sit nicely next to perlnavigator


Status?

Is this still something you are actively developing and bringing up to speed with the official Version 1 release of SQRL that was just published?

yes the idea to catch up with all the progress since initial sqrl discussions

I'd suggest, starting with a more up to date implementation.  I'm unable to contact the fork owner, but working from code that has already been updated to python 3, and a later version of django, might be a worthwhile endeavor.
Also, please add this to PyPi.  That's vital for anyone to use it, since it allows easy upgrading of the library.  (It's not hard
https://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/creation.html
https://git.kiichan.space/kiichan/django-sqrl-2


Degree filter on large query

Does this PR close any open issues?
Closes #
Give a longer description of what this PR addresses and why it's needed
At the moment we don't have any way to display results from queries that are > 100.
Adding a check to see how large the query result is will trigger a filtered matrix.
Provide pictures/videos of the behavior before and after these changes (optional)
Are there any additional TODOs before this PR is ready to go?
TODOs:

[ ] Find a semantically meaningful min degree value
[ ] Change the hover text for filter to show # of children

Netlify does not like how I am treating the hover object -- I tried to add a check for it but it didn't work. Line 167, MultiMatrix.vue

Test Query: RPC1,
Node 0 Label ~= CBB

Created issue #417 because although this solves larger query results, it is a not a solution for very large query returns that crash the browser.
Example query:
workspace: marclab
network: RC1---
query: node0.Label ~= CBB


Update tsconfig.json enable strict mode

there's still several strict errors, but we're closer to some with https://github.com/matrix-org/matrix-events-sdk/commit/f55466253407da3f77dd942a7b3c52671f685cbd#diff-b55cdbef4907b7045f32cc5360d48d262cca5f94062e353089f189f4460039e0

Achieved from:

https://github.com/matrix-org/matrix-events-sdk/commit/a811a16072c3a67d27a04fb730fed96b6cd57240
https://github.com/matrix-org/matrix-events-sdk/commit/0225a9e2a969e9bd16b17b39ffd5a15ad75f367a
https://github.com/matrix-org/matrix-events-sdk/commit/9567c7c570531da03171b21fd483312a729a74fa
https://github.com/matrix-org/matrix-events-sdk/commit/7de25f28b13c2b881301f174612b168a98d14a01
https://github.com/matrix-org/matrix-events-sdk/commit/2058391e31f2abec4d61b237ad53faa8e546a614


RUM-6129 Introduce the API usage telemetry event and API

What does this PR do?
In this PR we are introducing the new InternalLogger#logApisUsage method which will be used to report usage of different APIs as Telemetry from our sdk. In order to achieve this we had to refactor and simplify the whole Telemetry related logic in the SDK to make it more scalable. For this we introduced the new InternalTelemetryEvent which will be shared through out all the module from the dd-sdk-android-internal module.
Motivation
What inspired you to submit this pull request?
Additional Notes
Anything else we should know when reviewing?
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] Make sure you discussed the feature or bugfix with the maintaining team in an Issue
[ ] Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

Codecov Report
Attention: Patch coverage is 62.32877% with 55 lines in your changes missing coverage. Please review.

Project coverage is 69.96%. Comparing base (d059818) to head (94d452d).
Report is 7 commits behind head on develop.

Files with missing lines
Patch %
Lines

...ndroid/telemetry/internal/TelemetryEventHandler.kt
40.62%
36 Missing and 2 partials :warning:

...og/android/telemetry/internal/TelemetryEventExt.kt
0.00%
9 Missing :warning:

.../main/kotlin/com/datadog/android/_InternalProxy.kt
81.25%
0 Missing and 3 partials :warning:

...n/com/datadog/android/core/internal/DatadogCore.kt
81.82%
0 Missing and 2 partials :warning:

.../android/core/internal/logger/SdkInternalLogger.kt
94.44%
0 Missing and 1 partial :warning:

...lin/com/datadog/android/rum/internal/RumFeature.kt
88.89%
0 Missing and 1 partial :warning:

.../android/rum/internal/monitor/DatadogRumMonitor.kt
66.67%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2258      +/-   ##
===========================================
- Coverage    70.05%   69.96%   -0.09%
===========================================
  Files          730      727       -3
  Lines        27189    27104      -85
  Branches      4590     4570      -20
===========================================
- Hits         19045    18961      -84
- Misses        6842     6868      +26
+ Partials      1302     1275      -27

Files with missing lines
Coverage Δ

...n/kotlin/com/datadog/android/api/InternalLogger.kt
100.00% <ø> (ø)

...g/android/rum/internal/domain/scope/RumRawEvent.kt
100.00% <100.00%> (ø)

...dog/android/telemetry/internal/TelemetryEventId.kt
100.00% <100.00%> (ø)

...atadog/android/telemetry/internal/TelemetryType.kt
100.00% <100.00%> (ø)

.../android/core/internal/logger/SdkInternalLogger.kt
87.50% <94.44%> (-2.40%)
:arrow_down:

...lin/com/datadog/android/rum/internal/RumFeature.kt
93.65% <88.89%> (+0.05%)
:arrow_up:

.../android/rum/internal/monitor/DatadogRumMonitor.kt
84.90% <66.67%> (-2.67%)
:arrow_down:

...n/com/datadog/android/core/internal/DatadogCore.kt
82.43% <81.82%> (+0.53%)
:arrow_up:

.../main/kotlin/com/datadog/android/_InternalProxy.kt
69.23% <81.25%> (+5.59%)
:arrow_up:

...og/android/telemetry/internal/TelemetryEventExt.kt
75.00% <0.00%> (-25.00%)
:arrow_down:

... and 1 more

... and 35 files with indirect coverage changes


Update Alpine to 3.5

https://hub.docker.com/_/alpine/

See https://github.com/docker-library/python/pull/166, https://github.com/docker-library/python/pull/168, and https://github.com/docker-library/golang/pull/131#issuecomment-270677556.

:'-(
May be add
3.6.0-alpine, 3.6-alpine, 3-alpine, alpine (3.6/alpine/Dockerfile) / stay with 3.4 Alpine for now, add deprecation warning and switch to latest alpine later and after that always use latest alpine /
3.6.0-alpine3.4, 3.6-alpine3.4, 3-alpine3.4, alpine3.4 (3.6/alpine/Dockerfile)
3.6.0-alpine3.5, 3.6-alpine3.5, 3-alpine3.5, alpine3.5 (3.6/alpine/Dockerfile)

I would also use a python:3.6-alpine3.5 tag if there was one (driven mostly by LibreSSL, postgres @ 9.6, and no warning during update-ca-certs) ✨

Plz update to alpine 3.5

The docker-library/golang repository included the -alpine3.5 variant (https://github.com/docker-library/golang/tree/master/1.7/alpine3.5) following the discussion in https://github.com/docker-library/golang/pull/131.
How about using the same approach here?

It was a big mistake not to go for Alpine 3.5 when Python 3.6 was released.

@Lubo Python 3.6 was released 2016-12-23; Alpine 3.5 was released 2016-12-22 -- we didn't even know we'd have to be reverting anything at that point, but yeah, we should've left 3.6 (a bit late now)
@nigma yeah, that's sane -- once https://github.com/docker-library/python/pull/194 settles, I'll take a look at adding alpine3.5 variants here

Trying to use Python 2.7 with R, which only available in alpine 3.5. Any reasons not to upgrade?

Alpine 3.6 is out a it's the new stable. Shouldn't we jump right there? Also, this issue has been around for nearly half a year now. What can we do speed the things up?

I've created PR #201 in order to bump the Alpine version to 3.6. The build is green for the PR.
I'm happy to reroll it to introduce a alpine3.6 tag if that is the preferred approach.

Yes, let's add explicit alpine3.6 variants here.  See https://github.com/docker-library/golang/pull/162 for an example of doing that for the golang image.

I think we should start conservative and only include python:3.6 and possibly python:2.7 initially (to also help encourage folks to update to the latest version).

Also, let me know if you are no longer interested in taking the lead on this change and I'm happy to take over.

@tianon thanks for the offer. I like a challenge so I'd like to continue to work on this. I would appreciate your feedback when I push my changes. I hope to get that done later today or some time tomorrow Australian time.

@tianon it took me a little long than planned, but I have made the changes you suggested above. There is an issue with the build that I would appreciate some help with. See PR #201 for all the details.

@tianon now that PR #201, which adds alpine 3.6 support for Python 2.7 and 3.6, has been merged can we close this issue?

Yep, thanks!


Added new dz! macros in dataframe.rs to create dataframe from vector of mixed data type vectors

Enhancement - a new macro dz! to create ECharts dataframe from vector of mixed data type vectors

To close https://github.com/yuankunzhang/charming/issues/69


Broken link to DPS docs in IoT Central applications

Dear IoT Central team,
I didn't know where to report this, so I hope you don't mind I'm doing it in this page, which is unrelated I think 😮
On the IoT Central application, under "Permissions" → "Device connection groups", in the following section...

Clicking on the blue "Learn more" link returns a 404 error code.
I suppose this link should lead to the IoT Hub DPS documentation instead 😄
On another note, congrats for the great work so far!
Keep it up!!!
Jaime

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 829b0d23-80b6-10ac-a919-9e155357b42b
Version Independent ID: 5fd62458-72a8-d223-ab21-20ed85b7b882
Content: What is Azure IoT Central
Content Source: articles/iot-central/core/overview-iot-central.md
Service: iot-central
GitHub Login: @dominicbetts
Microsoft Alias: dobett

@chronoclast Thanks for the feedback! We are currently investigating and will update you shortly.

@chronoclast We have reported it internally and ensure that we will update with the correct link.
Thanks again for reporting this.
On another note, congrats for the great work so far!
Thanks for your valuable feedback!!!
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


docker-machine optional for linux

@arun-gupta
Setup instructions should specify that docker-machine is optional for linux.

Docker Machine is required to create Docker Swarm clusters and so is required.


Improve logging

We should use the logging library (instead of using print as we do now), and ideally we would show the error when parsing fails.

Closed by https://github.com/eigenfoo/cryptics/commit/28cd4ccb2cf4cf940868bbe6bad4b8e1b8d8bfbe


Add support to run Spark interpreter on a Kubernetes cluster

What is this PR for?
The goal of this PR is to be able to execute Spark notebooks on Kubernetes in cluster mode, so that the Spark Driver runs inside Kubernetes cluster - based on https://github.com/apache-spark-on-k8s/spark. Zeppelin uses spark-submit to start RemoteInterpreterServer which is able to execute notebooks on Spark. Kubernetes specific spark-submit parameters like driver, executor, init container, shuffle images should be set  in SPARK_SUBMIT_OPTIONS environment variable. In case the Spark interpreter is configured with a K8 Spark specific master url (k8s://https....) RemoteInterpreterServer is launched inside a Spark driver pod on Kubernetes, thus Zeppelin server it has to be able to connect to the remote server. In a Kubernetes cluster the best solution for this is creating a K8S service for RemoteInterpreterServer. This is the reason for having the SparkK8RemoteInterpreterManagerProcess - extending functionality of RemoteInterpreterManagerProcess - which creates the Kubernetes service, mapping the port of  RemoteInterpreterServer in Driver pod and connects to this service once Spark Driver pod is in Running state.
Design considerations: As described in spark-interpreter-k8s.md, the Zeppelin Server is running inside the Kubenetes cluster - thus we can choose where to run the Zeppelin server - the benefit of running the server inside K8S is that we don't have to deal with authentication. However is not enough to start only the Zeppelin Server inside the Kubernetes cluster as by default Zeppelin will start spark-submit in the same pod and will run every Spark job locally. The scope of this PR is run to run spark-submit (apache-spark-on-k8s version) properly configured with Docker images etc. so that the Spark driver will be started in a separate pod in the cluster, also staring separate pods for Spark executors  thus we can benefit from dynamic scaling of executors inside the Kubernetes cluster (while all the scheduling, pod allocation, resource management is done by the Kubernetes scheduler).
Please see below how is this running/used:
The cluster:

The flow:

What type of PR is it?
Feature
What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-3020

How should this be tested?
Unit and functional tests - running notebooks on Spark on K8S.
Questions:

Does the licenses files need update?
Is there breaking changes for older versions?
Does this needs documentation?

Thanks @matyix for this contribution, could you add some doc to illustrate how to use this feature in zeppelin ?

Hello @zjffdu @felixcheung- I have added docs about how to use/build this PR as well. Also the Dockerfile has been removed, the documentation contains details of using a third party (mine) or build your own Dockerfile - it was added in the repo as a convenience, but due to licensing concerns I did remove it. There was no particular reason why it was 1.7 (at the time I started to work on this 1.8 was not released yet - and I usually waiting for the first patch version until I switch).

@zjffdu @felixcheung I have updated the original PR description with design considerations and a typical K8S cluster and Zeppelin flow showing how I am using this feature/PR currently on the https://github.com/apache-spark-on-k8s/spark Spark version.

@matyix I have a local spark-k8s setup and have this branch (without success so far, so debugging SparkK8RemoteInterpreterManagedProcess to tackle down the issue). A few questions:

I see that in case of 'k8s://...' master, the 'SparkK8RemoteInterpreterManagedProcess' is first search for a driver pod starting with prefix zri- - Does that mean I have to manually instanciate that pod?
If created one such pod, and for now, if the pod is found, the 'SparkK8RemoteInterpreterManagedProcess' considers at as "remote process" - Actually, you don't create a process via the classical zeppelin 'interpreter.sh' command. Right?
Then, I receive java.net.ConnectException: Connection refused.

Would the classical spark-submit via interpreter.sh not be another option? Did you try it? Here, using that approach, the driver is created but exits directly. I guess https://github.com/apache-spark-on-k8s/spark/pull/402 would help.

Just tried the vanilla (so without this #2637 PR) zeppelin on spark-k8s with the in-cluster client mode branch (https://github.com/apache-spark-on-k8s/spark/pull/456).
It works fine out-of-the-box (with the ad-hoc spark interpreter parameters).
@matyix Do you see any reason to further work on this PR? Maybe you want to address additional goals with this?

@echarles @zjffdu @felixcheung
It is absolutely makes sense to keep this PR and make further work with it. Just to re-empahsize, the goal is to enable Zeppelin to submit notebooks to a Kubernetes cluster invoking spark-submit in cluster deploy mode.
Please find below a couple of advantages the cluster mode has comparing to the client mode:

currently the cluster client mode appears to have some problems - I faced exactly the same problems what you have  described in the PR when running multiple interpreters and I’m not sure if
whether these problems will be resolved and client mode will be supported (I have some PR's on Spark-k8s fork and will catch up with the folks regarding this topic)
in cluster mode you are running Zeppelin server and each RemoteInterpreterServer process (Spark Driver) is running in separate pods which fits better to Kubernetes best practices/patterns (instead of having one monolith RIS)
the latest Spark Driver creates a separate K8S Service to handle Executor --> Driver connections in cluster mode, which again fits better in Kubernetes best practices/patterns
this solution works regardless of Zeppelin server running in/outside of cluster if we add the option to set up authentication info for Zeppelin
it is using spark-submit and interpreter.sh and simplifies a bit the spark-submit command for K8S. Other than this the PR created  SparkK8RemoteInterpreterManagedProcess to simplify connection to RemoteInterpreterServer in K8S clusters, so we are using K8S client to look up the Driver pod then create a K8S Service is bounded to RemoteInterpreterServer running inside Driver pod
overall this may seem a bit more complicated then client mode however it works better and fits better in Kubernetes cluster best practices/patterns
if you have some ideas about a better way to place this functionality in Zeppelin, please let me know

Overall this is a way better and cleaner approach which fits the K8S ecosystem and at the same time has no side-effect for those not willing to use K8S.
I will update the PR regardless to fix the merge conflicts and add some minor changes/improvements - I am using this PR extensively on a few large K8S clusters and it works/fits our needs on K8S and complies with the our K8S cluster standards/best practices.

@matyix I have tested your last commits and was able to make it work in my env (with both zeppelin in- and out k8s cluster).
Your implement a new (specific for spark-k8s) launch and remote executor. In another local branch, I have tried to stick as much as possible to the current zeppelin paradigm (thrift servers both sides of the interpreters processes with CallbackInfo) and 2 parameters (host, port) for interpreter.sh - I still have issue with the callback, so I finally think the approach you propose is good and does the job.
My feedbacks:

The branch as-such need basic updates: I had to fix compilation issue with the new classes (SparkK8sInterpreterLauncher and SparkK8sRemoteIntepreterManagedProcess) and had to add ${ZEPPELIN_SPARK_CONF} in the interpreter.sh script.
To find the running driver pod, you actually poll on regular basis. The ideal would be to be notified when the pod is ready (not sure if the k8s client support this. We would closely map the current mechanism of the thrift notification via the CallbackInfo, but here with a pure k8s mechanism. This could be also extended to other interpreters we would want to see in k8s.
We need to set spark.app.name with must a value starting with zri- - If you don't set this in the interpreter settings, the k8s client will not find the driver pod - I wonder if we can make this more configurable, let's say using metada or simply using the InterpreterContext with contains a properties attributes with all the given props - the launcher could retrieve this and search for a pod starting with a dynamic prefix rather than with this hardcoded one.
The current vanilla zeppelin supports out-of-the-box the spark-k8s client-mode (assuming you are using https://github.com/apache/zeppelin/pull/2637). The condition to use the SparkK8sInterpreterLauncher needs to check for spark.submit.deployMode being cluster and continue to use the normal ManagedProcess for client.
On documentation level, certainly mention that the app name must start with zri-. Also, relying on the kubespark docker image would be better to ensure nothing special is added in the docker image.

WDYT?
Do you prefer me to submit a PR on your PR and will you make another push?

PS1: I have pushed my fixes in zeppelin-k8s/tree/spark-interpreter-k8s-fixes branch (which has merge conflict with master due to latest commit 3b1a03f that touch the launchers and remote executors)
PS2: I have opened on PR to document this on the spark-k8s docs repo apache-spark-on-k8s/userdocs/pull/21

Hello @echarles, thanks for the feedback, followed up on that and made the suggested changes, please see below:

use watcher instead of polling
set driver prefix form config properties spark.app.name
use sparkK8Launcher only if deploy-mode=cluster
no updates needed to interpreter.sh
added groupId to InterpreterLaunchContext

The branch can be merged as of now - unluckily the master is moving fast. I've seen your comment/branch a bit late  but pretty much made the same changes (like fixing the build).

@zjffdu Spark submit needs a resource staging server (RSS) to be specified as a parameter, you should start RSS, get the address of RSS - as described in the documentation which is part of the PR - https://github.com/banzaicloud/zeppelin/blob/spark-interpreter-k8s/docs/interpreter/spark-interpreter-k8s.md - and specify these:
--deploy-mode cluster --kubernetes-namespace default
--conf spark.kubernetes.resourceStagingServer.uri=http://{RESOURCE_STAGING_SERVER_ADDRESS}:10000
--conf spark.kubernetes.resourceStagingServer.internal.uri=http://{RESOURCE_STAGING_SERVER_ADDRESS}:10000

Thanks @matyix Looks like the doc here need to be updated. https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html

@zjffdu Not sure, never followed that one but I'll check with the folks over there as I have a few PR's on that repo as well.

Tested latest commit on AWS K8S cluster and works great as well in client as cluster modes - Kudos @matyix

Thanks @echarles. Let me know if anything else is needed to get this merged.

@matyix If my tests are correct, for now, we can not set spark.app.name and spark.kubernetes.driver.pod.name (if you set them, the driver will not be found by zeppelin).
Upon that is not user-friendly, the side effect is that pod driver and executors names start with null-....
IMHO it is a matter of passing the correct parameters to spark-submit depending on the presence or not of those properties.

Hello @echarles
You can set both spark.app.name and spark.kubernetes.driver.pod.name from interpreter settings and they will be set for spark-submit however you’re right regarding checking the driver pod name, I’m expecting a specific prefix. This can be changed to check whether there’s a specific pod name set for the interpreter in settings. Let me know and I can go ahead with the change.

I am playing in cluster mode setting/removing via the interpreter page the spark.app.name and spark.kubernetes.driver.pod.name and so far the only to make it work (= remote executor finding the spark driver) is really having the absence of those props in the interpreter page (maybe I am fooled with  spark.app.name).
Yes, checking if a specific spark.kubernetes.driver.pod.name  prop and using it will be an expected enhancement, and may even solve the  spark.app.name potential issue.
It also seems to me that Zeppelin adds spark.app.name and master props if they are not present.
I guess we should for now rely on that behavior and ensure in this PR that enough intuitive and documented material is available for the end-user.

@matyix I made a few more tests, and now spark.app.name and spark.kubernetes.driver.pod.name can be set without problem... (to be further confirmed).

@echarles Added some nice to have features like a separate log4j config for k8s (same as for yarn), updated the doc and now I’m only checking the processId label on driver pod so users can freely change the driver pod name.

Thx @matyix. I will test and give feedback.
Did you adding external dependencies (via the interpreter page). It works on my setup in client mode but fails (ClassNotFound) in cluster mode. I can debug but your tests on this can help.

@echarles Currently there are two ways to add external dependencies: add a new paragraph to the notebook using spark.dep interpreter and z.load(). This works because it downloads the dependencies inside the driver. There is an issue with this on Spark 2.2 and Scala 2.11.8 similar to https://issues.apache.org/jira/browse/ZEPPELIN-2475. Adding artifacts to Interpreter setting doesn’t work in case of Spark cluster mode, since dependencies are downloaded locally. Maybe we can think about something like: in case of deployMode=cluster add artifact to --packages automatically and don’t download it locally.

@matyix There is a long history in Zeppelin on spark.dep vs external dependency in interpreter settings. I am fan of the later (interpreter settings), so if the --packages flag can make it work, this would be wonderful.
I don't see in Spark doc that --package add the jars on the executor classpath.
The spark.jars  property (Comma-separated list of local jars to include on the driver and executor classpaths) may be an alternative.

The packages option for spark submit is described here and it seems to work. Using this option seems to be better alternative vs. then spark.jars because makes more sense to download jars right inside driver and executor pods, where they will be used. I think we may address this in a separate PR because this could be useful for yarn cluster mode as well not just for k8s.

@matyix sure, we can to address the dep in separate PR, especially if it is beneficial for all deploy modes. The downside of it is that we will have everybody happy with any change on deps management (if you search the mailing list, you will a lot of questions and discussions around this).
An additional parameter is the definition of "local" dependencies that don't come from any maven repo on the Internet. I regularly have to define local jars on the local disk file system (or even resources files that are not jar) and with the current yarn-client behavior of Zeppelin (probably not for yarn-cluster, I didn't test), those jars/resources are available on the classpath (of the driver and executor).
If we could inject this now while this branch is being discussed, that would be really a good thing.

@echarles jars from Zeppelin local-repo are set for spark-submit in sparks.jars parameter

@matyix I've given a try on you last commit and can not get the additional deps (in settings page) working.
I don't see the spark.jars property in the generate command (by interpreter.sh):
/opt/spark/bin/spark-submit --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer --driver-class-path ":/opt/zeppelin/interpreter/spark/*:/opt/zeppelin/lib/interpreter/*::/opt/zeppelin/interpreter/spark/zeppelin-spark_2.11-0.8.0-SNAPSHOT.jar:/etc/hdfs-k8s/conf" --driver-java-options " -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///opt/zeppelin/conf/log4j.properties -Dzeppelin.log.file=/opt/zeppelin/logs/zeppelin-interpreter---zeppelin-k8s-hdfs-locality-zeppelin-7cd554b49d-dpq2k.log" --master k8s://https://kubernetes:443 --conf spark.cores.max='1' --conf spark.shuffle.service.enabled='false' --conf spark.yarn.dist.archives=/opt/spark/R/lib/sparkr.zip --conf spark.executor.instances='3' --conf spark.sql.catalogImplementation='in-memory' --conf spark.app.name='zeppelin-k8s-spark' --conf spark.executor.memory='1g' --conf spark.master='k8s://https://kubernetes:443' --conf spark.kubernetes.namespace='default' --conf spark.kubernetes.executor.docker.image='datalayer/spark-k8s-executor:2.2.0-0.5.0' --conf spark.kubernetes.driver.docker.image='datalayer/spark-k8s-driver:2.2.0-0.5.0' --conf spark.kubernetes.initcontainer.docker.image='datalayer/spark-k8s-init:2.2.0-0.5.0' --conf spark.kubernetes.resourceStagingServer.uri='http://<IP_ADDRESS>:10000' --conf

From the command line it seems that the default SparkInterpreter is launched not the K8 specific one. Could you please check if spark.submit.deployMode is set to cluster since thats the other condition to launch SparkK8IntrepreterLauncher?

Hi @naveenkumargp, there was a refactor around the interpreter packaging which caused the ClassNotFound problems. Previously there was a big jar containing the interpreter class as well, which doesn't exists anymore. We've updated the PR, so that all jar files from local repo are enumerated with --jar option to spark-submit, which probably is a better approach.
With regards to deploy mode: just tried to add spark.submit.deployMode property via UI and it worked for me,  we do set this by default to cluster in our custom interpreter.json, as well as the master property to k8s://https://kubernetes:443 however intentionally didn't included in the patch, since this is only optional.
We are using this patch about a half year now, in our deployments and works fine for us, would be glad to contribute to community. Please try this latest version if works for you.

Hi @sancyx
thanks for the update ,so now if we want to use this PR how to use. like should we take clone latest zeppelin source code add your changes manually and build.  OR is there any other git uri is there, for this particular commit  which includes this #PR. how it will be?
regards
naveen

@naveenkumargp Ideally, this should be merged (chances are unlikely if you check the history of the PR, and in a few days this will conflict with the master branch, so will be even more unlikely) otherwise use the fork we maintain and build from there: https://github.com/banzaicloud/zeppelin/tree/spark-interpreter-k8s and use the instructions from  https://banzaicloud.com/blog/zeppelin-spark-k8/ blog series.
We will be launching this as a service late May so you can get the platform and have it done all these above for you: https://github.com/banzaicloud

@sancyx
we tried by taking the source code of commit id 7f237809af7e62da75ecba26d4194c48e6e4cdd0  by building it  without modifying interpreter.sh, when we submit the spark job we are getting the following error when spark driver is been launched.
Error: Could not find or load main class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer
regards
Naveen

@naveenkumargp All jars from ZEPPELIN_HOME/lib/interepter should be added with --jar option. Could you please attach the generated spark-submit command in Zeppelin logs?

@sancyx
we have tried by adding --jars option interpreter.sh still we are getting following when the spark job is being submitted.
java.lang.RuntimeException: Unable to start SparkK8RemoteInterpreterManagedProcess: Spark Driver not found.
and the spark command which is being generated is given below
/opt/spark/bin/spark-submit --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer --driver-class-path ":/opt/zeppelin-0.9.0-SNAPSHOT/interpreter/spark/:/opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/::/opt/zeppelin-0.9.0-SNAPSHOT/interpreter/spark/spark-interpreter-0.9.0-SNAPSHOT.jar:/opt/hadoop/etc/hadoop/" --driver-java-options " -Dfile.encoding=UTF-8 -Dlog4j.configuration=log4j_k8_cluster.properties" --conf spark.executor.instances=2 --jars /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/aether-api-1.12.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/aether-connector-file-1.12.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/aether-connector-wagon-1.12.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/aether-impl-1.12.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/aether-spi-1.12.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/aether-util-1.12.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/bcpkix-jdk15on-1.52.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/bcprov-jdk15on-1.52.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/commons-codec-1.5.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/commons-configuration-1.9.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/commons-exec-1.3.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/commons-httpclient-3.1.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/commons-io-2.4.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/commons-lang-2.5.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/commons-logging-1.1.1.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/commons-pool2-2.3.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/gson-2.2.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/gson-extras-0.2.1.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/httpclient-4.5.1.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/httpcore-4.4.1.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/jline-2.12.1.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/jsoup-1.6.1.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/libthrift-0.9.2.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/log4j-1.2.17.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/maven-aether-provider-3.0.3.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/maven-artifact-3.0.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/maven-model-3.0.3.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/maven-model-builder-3.0.3.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/maven-plugin-api-3.0.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/maven-repository-metadata-3.0.3.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/plexus-classworlds-2.4.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/plexus-component-annotations-1.5.5.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/plexus-interpolation-1.14.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/plexus-utils-2.0.7.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/sisu-guice-3.0.2-no_aop.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/sisu-inject-bean-2.2.2.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/sisu-inject-plexus-2.2.2.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/slf4j-api-1.7.10.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/slf4j-log4j12-1.7.10.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/wagon-http-1.0.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/wagon-http-lightweight-1.0.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/wagon-http-shared-1.0.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/wagon-provider-api-1.0.jar /opt/zeppelin-0.9.0-SNAPSHOT/lib/interpreter/zeppelin-interpreter-0.9.0-SNAPSHOT.jar --master k8s://https://k8s-apiserver.bcmt.cluster.local:8443 --conf spark.submit.deployMode='cluster' --conf spark.app.name='Zeppelin-spark-shared-process' --conf spark.kubernetes.driver.label.interpreter-processId='spark-shared-process-1526377697510' --conf spark.metrics.namespace='Zeppelin-spark-shared-process' /opt/zeppelin-0.9.0-SNAPSHOT/interpreter/spark/spark-interpreter-0.9.0-SNAPSHOT.jar 30000

@matyix
we forked the your banzai cloud zeppelin code from uri https://github.com/banzaicloud/zeppelin/tree/spark-interpreter-k8s and tried building while building we are getting the following issue.
npm ERR! Linux 3.10.0-327.28.2.el7.x86_64
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "build:dist" "--https-proxy=http://<IP_ADDRESS>:8080" "--proxy=http://<IP_ADDRESS>:8080" "-verbose"
npm ERR! node v6.9.4
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>build:dist: npm-run-all prebuild && grunt pre-webpack-dist && webpack && grunt post-webpack-dist
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build:dist script 'npm-run-all prebuild && grunt pre-webpack-dist && webpack && grunt post-webpack-dist'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the zeppelin-web package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm-run-all prebuild && grunt pre-webpack-dist && webpack && grunt post-webpack-dist
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs zeppelin-web
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls zeppelin-web
npm ERR! There is likely additional logging output above.
npm verb exit [ 1, true ]
npm verb stack Error: Unknown system error -122: Unknown system error -122, open 'npm-debug.log.1139106934'
npm verb stack     at Error (native)
npm verb cwd /u/npanchap/banzaizeppelin/zeppelin/zeppelin-web
npm ERR! Linux 3.10.0-327.28.2.el7.x86_64
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "build:dist" "--https-proxy=http://<IP_ADDRESS>:8080" "--proxy=http://<IP_ADDRESS>:8080" "-verbose"
npm ERR! node v6.9.4
npm ERR! npm  v3.10.10
npm ERR! path npm-debug.log.1139106934
npm ERR! code Unknown system error -122
npm ERR! errno -122
npm ERR! syscall open
npm ERR! Unknown system error -122: Unknown system error -122, open 'npm-debug.log.1139106934'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     https://github.com/npm/npm/issues
npm verb exit [ -122, true ]
npm ERR! Please include the following file with any support request:
npm ERR!     /u/npanchap/banzaizeppelin/zeppelin/zeppelin-web/npm-debug.log

@naveenkumargp Not sure this is the best place discussing your problem ... anyways:
The message Unable to start SparkK8RemoteInterpreterManagedProcess: Spark Driver not found. means that Zeppelin was not able to find a running Spark Driver pod. I don't see Resource Staging Server and image params in your spark-submit command, these should be set in SPARK_SUBMIT_OPTIONS - checkout the description here:.
You can build an image from the Banzai Cloud branch, check out our .circleci/config.yaml or even better and easier, you may use/check Banzai Cloud's zeppelin-spark Helm chart to set up a working example on K8S.
I am not sure I can give more options than this - I might suggest reading/understanding how Spark/Zeppelin on Kubernetes works from our blog or just use the Pipeline platform ...

@matyix
Thanks for the information. The resource staging server and spark (driver/executor) images have been set in the spark-defaults.conf inside the zeppelin docker image. We believe that the issue as per our understanding is not because of that.
The class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer is not present in the spark-interpreter-0.9.0-SNAPSHOT.jar.
The class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer is present in zeppelin-interpreter-0.9.0-SNAPSHOT.jar.
Using this, is working for us. May be we are missing something.
The documentation and information in the blogs are very useful and informative.
If this is not the right forum for this discussion, can you please share where this can be discussed further.

@matyix @sancyx
when we had taken the snapshot it was missing the code related adding of jars from zConf.getZeppelinHome() + "/lib/interpreter".
with the latest code without any changes we are able to run zeppelin with spark on k8s .
thanks for all your help.

@naveenkumargp one more thing. There is another alternative we are experimenting - we are adding Kubernetes integration to Livy and you would be able to use the Livy interpreter (no Zeppelin code modification required) and spin up Spark on K8s (and you can use Jupyter as well). Let me know if you are interested - drop me a mail.

@matyix
yes we are very much interested on running Livy interpreter on K8s.
please let us know how we can proceed further.
regards
naveen

@matyix
Will this change be followed up for upstreaming or will it now be only with Livy.
With Livy, also it would be good to have this change also as a feature in Zeppelin.


Fix gan.py

add self.discriminator.zero_grad()
also return an additional argument errG from step method of GAN class

Thanks @abhi-glitchhg you can make the necessary changes in the GAN implementation


Migrate issues from SourceForge

Migrate any outstanding issues from SourceForge.

Bugs: http://sourceforge.net/p/supercsv/bugs
Feature Requests: http://sourceforge.net/p/supercsv/feature-requests/

@kbilsted I've migrated all the issues from SourceForge that I consider valid.  Take a look and let me know if there's any left you think should be copied across - otherwise I'll just close them all off.
I've also locked it down so no new bugs/feature requests can be raised via SourceForge, but I'll leave everything there for historical purposes (especially as the website release notes refer to those old issues).

great - how about
https://sourceforge.net/p/supercsv/bugs/52/
https://sourceforge.net/p/supercsv/feature-requests/42/
https://sourceforge.net/p/supercsv/feature-requests/33/ (2 issues.. one being a config for interning strings)
https://sourceforge.net/p/supercsv/feature-requests/29/
what do you think about including these?

I closed 33 (it should have been closed already as it was solved), and added the rest (including some others you didn't mention).
So now SourceForge will just be used for:

Historical purposes (release notes still point to old SF bugs/features)
Downloads (people are still going to download from SF, so may as well upload the distribution there as well...for now)
Donation URL (PayPal link breaks Maven site, so using SF donation URL as redirect still)
Redirecting old website URLs (old website pages redirect to GitHub pages so links on StackOverflow etc still work - this even includes the javadoc!)

great job!


attribute Google API requests to dask/dask-bigquery package

It would be helpful for me and my teammates at Google in the prioritization of our work on open source connectors to have requests initiated by Dask attributable to this connector. The way this is done is by adding a User-Agent header to requests containing the tool name and version.
In Python, the ClintInfo class(es) are used to set this header: https://googleapis.dev/python/google-api-core/latest/client_info.html
from google.api_core import client_info as rest_client_info
from google.api_core.gapic_v1 import client_info as grpc_client_info

bq_client_info = rest_client_info.ClientInfo(
    user_agent=f"dask-bigquery/{dask_bigquery.__version__}"
)
bqstorage_client_info = grpc_client_info.ClientInfo(
    user_agent=f"dask-bigquery/{dask_bigquery.__version__}"
)

with bigquery.Client(project_id, client_info=bq_client_info) as bq_client:
    bq_storage_client = bigquery_storage.BigQueryReadClient(
        credentials=bq_client._credentials,
        client_info=bqstorage_client_info,
    )
    yield bq_client, bq_storage_client
    bq_storage_client.transport.grpc_channel.close()

Examples with google.cloud.bigquery.Client:

ibis-bigquery
pandas-gbq
jupyter magics

Thanks for the suggestion and providing a nice code snippet @tswast! Setting those User-Agent headers sounds totally reasonable and we'll be sure to add them as a follow up to https://github.com/coiled/dask-bigquery/pull/4
cc @ncclementi


Support for WMS 1.3.0

For #25 and includes the patch in the PR for #312 and BOM handling from #256. All of the mods to wms.py should be in 1.1.1 and 1.3.0.
Changelog
WMS:

convert to the factory structure
updated handling for dimensions (this I haven't found a live service for, fyi)
expanded doctests for 1.3.0 and GetMap requests

util.py:

update openURL exception handling to include the vnd_* errors
update openURL exception handling for the additional namespaces
added strip_bom for the BOM patch

congrats, I've started similar work some time back, but you managed to get it together.
Question to all: does it make sense, before pull request is merged, to merge all commits into single commit? It's probably more easy for review. (git rebase -i COMMIT_HASH^)

@jachym +1.  @roomthily you can also do git rebase -i HEAD~33 (this PR has 33 commits).

Will try to get to it in the next day or two.
On Mon, Apr 25, 2016 at 4:27 AM, Tom Kralidis<EMAIL_ADDRESS>wrote:

@jachym https://github.com/jachym +1. @roomthily
https://github.com/roomthily you can also do git rebase -i HEAD~33
(this PR has 33 commits).
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/geopython/OWSLib/pull/323#issuecomment-214252135

--
Soren Scott
Research Coder
The Ronin Institute for Independent Scholarship
<EMAIL_ADDRESS>just a head's up - taking a bit of a sabbatical so if i don't
get back to you right away, no worries.

FYI I've looked into this deeper and it appears that the commits in this PR are over a long period of time, where commits happened in between.  It might be a bit messy to try and squash this using git rebase.
Having said this, I would suggest a clean branch based on current master with the WMS 1.3.0 based changes cleanly applied (some manual work probably needed).
@roomthily I'm willing to do this and will attribute the (much requested and awesome!) enhancement to you in the PR, but I want to make sure 1./ this is okay with you and 2./ you haven't begun this already

@roomthily any objections to my previous comment/question?

@roomthily bump.  Any objections/comments?

No, no all fine, was in agreement on the age of the commits and some funky history rewriting in rebase. Very clean PR fork in a few minutes.


Capabilities Can Be Declared More than Once

Minimal example:
(module j K
  (defcap K () true)
  (defcap K () false)
)

pact> (pact-version)
"3.0.1"
pact> (module j K (defcap K () true) (defcap K () false))
"Loaded module foo, hash QzMXwPQPm2qnSurrAoz-9nRPO2TZ0d2R3tHCTsPzMWQ"
pact>

Tested on master

dupe of #572


[backport 1.11] envoy: Bump cilium envoy to latest version v1.21.3

#20142 -- envoy: Bump cilium envoy to latest version v1.21.3 (@sayboras)

Once this PR is merged, you can update the PR labels via:
$ for pr in 20142; do contrib/backporting/set-labels.py $pr done 1.11; done

/test-backport-1.11

/test-backport-1.11

The CI failures are a bit suspect, are those passing on the v1.11 branch? :thinking:

https://jenkins.cilium.io/job/Cilium-PR-K8s-1.18-kernel-4.9/2083/testReport/junit/Suite-k8s-1/18/K8sPolicyTest_Basic_Test_Traffic_redirections_to_proxy_Tests_HTTP_proxy_visibility_without_policy/
Top 1 errors/warnings:
[[C15] cilium.network: Policy NOT FOUND for id: 9326 port: 80

I'm not quite sure what this means or why it only shows up in 1.11, but it looks like an Envoy issue.

seems like something goes wrong, all jobs are failing for below two jobs

Basic Test Traffic redirections to proxy Tests HTTP proxy visibility without policy
Basic Test Traffic redirections to proxy Tests proxy visibility interactions with policy lifecycle operations

Let me try to simulate locally

/test-1.16-netnext

/test-runtime

/test-runtime

/test-backport-1.11

Some failures are shown mainly due to /test command was used instead of /test-backport-1-11:

ci-l4lb --> success in https://github.com/cilium/cilium/actions/runs/2487971717
ci-awscni --> success in https://github.com/cilium/cilium/actions/runs/2487971724
test-1.24-4.19 --> not applicable as 1.24 run is not for v1.11
test-1.23-net-next -> 1.23 + net-next are not valid combination for v1.11. Job for 1.23 and 4.9 is successful here https://jenkins.cilium.io/job/Cilium-PR-K8s-1.23-kernel-4.9/196/
test-1.16-netnext -> hit https://github.com/cilium/cilium/issues/18889#issuecomment-1153989696


[THEME] Bootstrap Scrolling-nav

Implement the next theme:
https://startbootstrap.com/template-overviews/scrolling-nav/

Implemented, https://github.com/bludit/bludit/tree/master/bl-themes/alternative


telink: hal: update west.yml.

update west.yml for the changes of hal.

The following west manifest projects have been modified in this Pull Request:

Name
Old Revision
New Revision
Diff

hal_telink
https://github.com/telink-semi/hal_telink/commit/3cae467b96be07c142fa807bd3c7f900428d3674 (develop)
pull/87
<EMAIL_ADDRESS>

Note: This message is automatically posted and updated by the Manifest GitHub Action.


darwin/arm linker fails to handle pc-relative relocation to _cgo_yield

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
tip
What operating system and processor architecture are you using (go env)?
darwin/arm
What did you do?
The darwin/arm builder fails with relocation errors:
https://build.golang.org/log/8d0eeee7f56708b0f629533fab76cb27729f4229
What did you expect to see?
No relocation errors
What did you see instead?
runtime.notesleep: pc-relative relocation of external symbol is not supported
runtime.notesleep: unsupported obj reloc R_PCREL/4 to _cgo_yield
runtime.notesleep: pc-relative relocation of external symbol is not supported
runtime.notesleep: unsupported obj reloc R_PCREL/4 to _cgo_yield
runtime.notetsleep_internal: pc-relative relocation of external symbol is not supported
runtime.notetsleep_internal: unsupported obj reloc R_PCREL/4 to _cgo_yield
runtime.notetsleep_internal: pc-relative relocation of external symbol is not supported
runtime.notetsleep_internal: unsupported obj reloc R_PCREL/4 to _cgo_yield
runtime.notetsleep_internal: pc-relative relocation of external symbol is not supported
runtime.notetsleep_internal: unsupported obj reloc R_PCREL/4 to _cgo_yield
runtime.notetsleep_internal: pc-relative relocation of external symbol is not supported
runtime.notetsleep_internal: unsupported obj reloc R_PCREL/4 to _cgo_yield
runtime.findrunnable: pc-relative relocation of external symbol is not supported
runtime.findrunnable: unsupported obj reloc R_PCREL/4 to _cgo_yield
runtime.sysmon: pc-relative relocation of external symbol is not supported
runtime.sysmon: unsupported obj reloc R_PCREL/4 to _cgo_yield
CL 35494 introduced _cgo_yield which hit an unimplemented path in the darwin/arm linker. Darwin/arm64 is fine.

@crawshaw do you know enough about the darwin/arm linker to suggest a fix?


createTextEditorDecorationType with Theme Selection colors shows invisible text in all Themes but High-Contrast

VSCode Version: 1.30.0
OS Version: Windows

Steps to Reproduce:

Create an extension with the following files
package.json

{
    "name": "decorator-select",
    "description": "Decorator with Theme Selection colors",
    "publisher": "meme",
    "version": "1.0.0",
    "engines": {
        "vscode": "^1.30.0"
    },
    "activationEvents": [
        "onCommand:extension.decoratorselect"
    ],
    "contributes": {
        "commands": [
          {
            "command": "extension.decoratorselect",
            "title": "Decorator with Theme Selection colors"
          }
        ]
    },
    "main": "./index.js"
}

index.js
const vscode = require('vscode');

function activate(context) {
    var decorator = vscode.window.createTextEditorDecorationType( {
      color: new vscode.ThemeColor('editor.selectionForeground'),
      backgroundColor: new vscode.ThemeColor('editor.selectionBackground')
    });

    context.subscriptions.push(
        vscode.commands.registerCommand('extension.decoratorselect', () => {
            const editor = vscode.window.activeTextEditor;
            if (!editor) { return; }
            editor.setDecorations(decorator, [new vscode.Range(new vscode.Position(1, 0), new vscode.Position(1, 20))]);
        })
    );
};

function deactivate() {}

module.exports = {
    activate,
    deactivate
}

Load a file with some text (>30 chars) on line 2
Call command "Decorator with Theme Selection colors"
The decorated text has the foregroundColor equal to the backgroundColor. Except if you use the High Contrast Theme.

@rioj7 I think this is working properly. Here I have used your steps and additionally added the following settings:
"workbench.colorCustomizations": {
	"editor.selectionForeground": "#ff0000",
	"editor.selectionBackground": "#00ff00",
},

The correct colors appear to be picked up, the foreground is of the color editor.selectionForeground and the background is of the color editor.selectionBackground.
Do you perhaps find that you don't agree with the colors chosen in some of the built-in themes?

@alexandrudima I did a bit of experimenting with the themes.
Without color customisation
If you have a file, like JavaScript or HTML, with syntax highlighting and you select a piece of text only the background color changes except for the High Contrast theme where it uses the editor.selectionForeground color as defined in the theme.
Most likely this line is the cause of this
src/vs/platform/theme/common/colorRegistry.ts:307
export const editorSelectionForeground = registerColor('editor.selectionForeground', { light: null, dark: null, hc: '#000000' }, nls.localize('editorSelectionForeground', "Color of the selected text for high contrast."));

With decorator
The decorated text has the foreground and background color equal to the theme editor.selectionBackground. Only the High Contrast theme uses the theme editor.selectionForeground color.
With color customisation
Adding the following to the user level settings
"workbench.colorCustomizations": {
	"editor.selectionForeground": "#ff0000",
	"editor.selectionBackground": "#00ff00",
},

When you now select some text the same behavior except the editor.selectionBackground is now the same (green) for all themes. Only the High Contrast theme uses the set editor.selectionForeground defined in the user settings.
If I overrule the editor.selectionForeground I expect that every theme will use that as the foreground color for selected text.
With decorator
Every theme uses Red-on-Green for the decorated text
Changing the decorator
If I only set the background color in the decorator
    var decorator = vscode.window.createTextEditorDecorationType( {
      // color: new vscode.ThemeColor('editor.selectionForeground'),
      backgroundColor: new vscode.ThemeColor('editor.selectionBackground')
    });

I get the opposite behavior. For all themes except High Contrast, the decorated and selected text look the same. In High Contrast the decorated text is unreadable because all the color are bright.
Question
How do I construct a decorator that looks the same as selecting text for all the themes: Light, Dark or High Contrast?
When the editor.selectionForeground color is defined in the settings. Why is it not used in Light and Dark Themes?

When the editor.selectionForeground color is defined in the settings. Why is it not used in Light and Dark Themes

Enabling the selection forground color comes with a considerable performance penalty. That's why @alexandrudima decided to enable it only for high contrast themes. We could allow it for the themes that want it, but very likely the editor will get the blame for the slower performance, not the theme... :-(

How do I construct a decorator that looks the same as selecting text for all the themes: Light, Dark or High Contrast?

I agree.that's difficult. One trick would be to define a new color:
    "colors": [
      {
         "id": "fooExtension.selectionForeground",
        "defaults": {
          "highContrast": "editor.selectionForeground",
          "dark": "#0000",
          "light": "#0000"
        }
      }
    ],

Why do you want to create a decorator with the same colors?

I want to create a decorator that works out of the box with any theme. That means using colors defined in the theme.
I have used the following solution:

default the decorator only uses the editor.selectionBackground
have a setting to also use the editor.selectionForeground (only needed for high contrast themes)

My decorator only briefly decorates a piece of text so not likely that it will be mistaken for selected text.

What about my suggestion from https://github.com/microsoft/vscode/issues/78660#issuecomment-522893393 ?
It defines a new color that you can use. It defaults to editor.selectionForeground in hight contrast and to transparent for light and dark. That's the same behavior as the VSCode selection foreground.

@aeschli I had implemented my alternative already. I will try the custom color trick.
Thanks for the comments.

@aeschli I tried the trick with the custom colors but the effect is the same. The text color is equal to the background color in the light and dark themes.
If I use the Developer Tools I only see the following css rules that applies
.monaco-editor .ced-1-TextEditorDecorationType56-1 {
    color: rgba(0, 0, 0, 0) !important;
}
```
I can't find the method that sets the background of the decorator.

I have to go back to my original method:

For the background color I keep the custom color but I modify the foreground color of the Decorator based on the value of a setting.

Hm, I don't know why my 'trick' doesn't work, but it would take me some time to set up the full a code example. Do you have something that works?
In essence, the editor has a special rules how a selection looks for HC themes. For an extension to know how the selection looks like it would need to know the theme and the current color values. The request for this is #32813.
If that's fine with you, I close it as dup of #32813. If not, please summarise what API is needed instead or what changes should be done on our side.


Surface latest Redis dist caching package

Fixes #11118

Technically there's still references to AddDistributedRedisCache on the page. In the new package, it became AddStackExchangeRedisCache

@guardrex There's still references to AddDistributedRedisCache on the page. In the new package, it became AddStackExchangeRedisCache

I was hoping that more wouldn't have been required given that the sample is 2.1. Let me see if I can patch it up a bit more. I'm not going to upgrade the sample app at this time.

I didn't know there was a sample app. I was just talking about the small "Distributed Redis Cache" section on the page with some links and 4 lines of code.

I see it. I'm tracking down the API ref for it now. Should have it shortly.


OneDrive integration does not seem to work

seems to be working for me

How can we reproduce that bug @Amin-Dhouib ?

Try to use the<EMAIL_ADDRESS>email
(I will send the password to you in private)

@AminDhouib that's weird actually xD, @MohamedBechirMejri take a look

@AminDhouib it doesn't work only with your account ahhah xD

@AminDhouib @MohamedBechirMejri the problem is we don't really use that much outlook account, so we had to test this feature only with the default account<EMAIL_ADDRESS>


fix(router): context issues

It looks big, yes. But it boils down to:

Better encapsulation for the different states of the match loading process. That’s really just moving some code into functions like runBeforeLoad, etc
A change to use getMatch() and updateMatch exclusively, instead of reading and writing to a mix of local and state variables. This has already gotten rid of a few areas where we were definitely reading from stale data (but likely not getting any bugs reported)
Tracking and deduping promises for beforeLoad. This is a very important piece to the fix, since prior to this, we were skipping the before load stage if it was in progress and immediately running the loader. This tracking is what enables each load to dedupe and share in-flight promises for both beforeLoad, component loading, lazy loading, and ultimately loader calls.

fixes #1892
fixes #1886
fixes #1752
fixes #1752
fixes #1827
fixes #1534
fixes #1777
fixes #1814
fixes #1656
fixes #1796
fixes #1774
fixes #1751
fixes #1882
fixes #1795
fixes #1551
fixes #1928

@tannerlinsley This in the Transitioner was previously changed from useLayoutEffect to useEffect in an attempt to fix the context issues. Here: https://github.com/TanStack/router/pull/1713/files#diff-8f39ee7f0cd19573624baf67df23085405045c8e446263697011394594e56e17R32
For the function of subscribing to the current location, should this go back to being a useLayoutEffect? or remain as a useEffect?

Great to see this! 🌟
Does this also aim to address #1795?

@tannerlinsley This in the Transitioner was previously changed from useLayoutEffect to useEffect in an attempt to fix the context issues. Here: https://github.com/TanStack/router/pull/1713/files#diff-8f39ee7f0cd19573624baf67df23085405045c8e446263697011394594e56e17R32
For the function of subscribing to the current location, should this go back to being a useLayoutEffect? or remain as a useEffect?

I think we should subscribe as soon as possible. So even if we need to split them up, let's do that.

This looks pretty good! One thing I am still struggling to understand is how does notFound() work @tannerlinsley? I have this route.tsx file which isn't an actual path, it's a scaffolding for other matched routes.
import { createFileRoute, notFound } from '@tanstack/react-router'

export const Route = createFileRoute('/_loggedin/_layout/_adminuser')({
  beforeLoad: async ({ context }) => {
    const canShow = await context.checkIfCanShow()

    if (!canShow) {
      throw notFound()
    }
  },
  component: () =>
A
,
  errorComponent: () =>
B
,
  notFoundComponent: () =>
C
,
  pendingComponent: () =>
D
,
})

I would expect/like to render "C" when notFound() is thrown, but this doesn't work. In the older versions (1.33.x), actually nothing would get rendered, the whole component tree was gone (😳). In this pull request, I see the default Not Found text and the warning

A notFoundError was encountered on the route with ID "root", but a notFoundComponent option was not configured, nor was a router level defaultNotFoundComponent configured

If I comment out the throw notFound() part, I see "A" rendered as expected. Any hints?

@mrlubos if you throw a notFound in a beforeLoad, only a single not found component can be shown.
Currently, this PR will always invoke the root not found component and not the route's one.
@tannerlinsley told me we could change this to show the route's not found component. Could we also make it bubble up so it chooses the nearest not found component in case the route, whose beforeLoad had thrown the notFound, does not have a not found component?

I'd like that @schiller-manuel. I will admit that I still have knowledge gaps around how layout routes are matched as my understanding is they're not a part of the same tree as regular routes and I thought that might be causing the problem here. Ultimately my goal is to be able to throw and handle from within the same component (I tried using the routeId option in notFound() but it made no difference). I have a layout component I'm using to check permissions for certain user type and if the user doesn't have sufficient permissions, they should be shown the 404 page without revealing the existence of specific routes.

This pull request can't be released fast enough. I discovered another issue introduced in 1.36.3 where a simple <Link to="/path" /> keeps reverting me back to the current path. Unfortunately, it's not obvious why or how to debug, but this problem didn't exist prior to 1.36.3, is present in the latest 1.44.4, and I can confirm is resolved in this pull request 🎉

This pull request can't be released fast enough. I discovered another issue introduced in 1.36.3 where a simple <Link to="/path" /> keeps reverting me back to the current path. Unfortunately, it's not obvious why or how to debug, but this problem didn't exist prior to 1.36.3, is present in the latest 1.44.4, and I can confirm is resolved in this pull request 🎉

thats sounds very similar to problems I am experiencing too! Will do a one time bonus donation if this big PR gets done 😃 ! thnks!

@tannerlinsley there is currently a bug that causes routes with loaders to sometimes (?) keep pending
I forked the basic-file-based example and installed the package of this PR:
https://stackblitz.com/edit/tanstack-router-2zgjft

click on posts
click on various posts
some load and display, others keep the previous post displayed or just show a blank page

https://github.com/user-attachments/assets/8cef0772-568d-4c56-8d11-2a58b9d63424

@schiller-manuel i also noticed my app became way less performant, but as always hard to say why, I didn't dig into the internals

i also noticed my app became way less performant, but as always hard to say why, I didn't dig into the internals

I've noticed that with intent preloading enabled, my application hangs very noticeably during the preload (hover over a link).

i also noticed my app became way less performant, but as always hard to say why, I didn't dig into the internals

I've noticed that with intent preloading enabled, my application hangs very noticeably during the preload (hover over a link).

I have the same degraded performance outcome with preload enabled.

@ruisaraiva19 can you please provide a minimal complete example by forking one of the existing examples on Stackblitz that shows this performance degradation?

@ruisaraiva19 can you please provide a minimal complete example by forking one of the existing examples on Stackblitz that shows this performance degradation?

Yes, I'll do it later today. Just to be clear, the performance issue happens specifically when hovering links (triggering the preloading)

Nice find, I believe it's the same for me, I use the same preload setting

I can confirm performance degradation after running a snapshot of this PR too.
Will wait for @ruisaraiva19 example to provide an example for this.
In the meanwhile here are some flame graphs to showcase how much the degradation in our case:

(Ignore the entries generated by browser addons - it doesn't look much better without them.)

I can confirm performance degradation after running a snapshot of this PR too. Will wait for @ruisaraiva19 example to provide an example for this. In the meanwhile here are some flame graphs to showcase how much the degradation is in our case:
Old and fast:
New and slow:
(Ignore the entries generated by browser addons - it doesn't look much better without them.)

I’m seeing some function calls that I recently removed in the latest version of this PR. Is this up to date?

@schiller-manuel @tannerlinsley Here is a bare minimum demo based on the kitchen-sink-react-query-file-based example. The Dashboard index route is the one with the code changes.
A couple of notes regarding the demo:

At first, I couldn't recreate the performance issue with a bare kitchen-sink-react-query-file-based example (maybe too simple of an app?).
After debugging with a couple of console logs + the React Profile flame graphs, I noticed that route components are rerendered multiple times when hovering links.

With defaultPreload: false, route components would render 5 times on the first hover event and 3 times on the following hover events.
With defaultPreload: 'intent', route components would render 7 times on the first hover event and 5 times on the following hover events.
So with defaultPreload: 'intent', we get a few extra rerenders.
This can be confirmed by looking at the console logs when inside the Dashboard index route.

My experience with the performance issue happens on a web app with complex dashboards.

To confirm this with a minimal demo, I created a SlowComponent component that takes at least 5ms to render to simulate a complex component.
After rendering a few component instances, I reproduced the same performance issue when hovering over links like on the web app mentioned above. You can check that when hovering over a link, the color change (from blue to red) is not responsive.

Then, I wrapped the DashboardIndexComponent component in a React.memo and the rerender performance issue is gone since the route component does not rerender on the hover events anymore.

I tried the React.memo approach on the web app mentioned above and it fixed the issue as well which uses version 1.43.10 of the router.

To summarize, it seems that when hovering over links the router is rerendering pretty much all Outlet components, hence the React.memo approach to fix the issue.
Please let me know if any of my conclusions aren't correct and feel free to ask any questions!

Good find. I can take a look at these perf regressions later tonight.

I’m seeing some function calls that I recently removed in the latest version of this PR. Is this up to date?

I just remembered i pinned a certain commit.
Will redo this without pinning a certain commit but pinning the PR today or tomorrow.

Made another comparison with our app now without the addons.
Result is pretty much the same in terms of the factor of the performance degradation. The addons just add overall.
It's still a performance degradation compared to the old version from last months npm update cycle we did,
even with the newer commit.
1.34.9:

commit 64a8bdd:

Sweet. I’ll take a look. I already have a good idea of where to start.
Tanner Linsley
TanStack LLC
On Jul 14, 2024 at 12:17 PM -0600, martylukyy @.***>, wrote:

Made another comparison with our app now without the addons.
Result is pretty much the same in terms of the factor of the performance degradation. The addons just add overall.
It's still a performance degradation compared to the old version from last months npm update cycle we did,
even with the newer commit.
1.34.9:
oldandfast2.png (view on web)
commit 64a8bdd:
newandslow2.png (view on web)
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

I just pushed a few fixes to the MatchInner reactivity. Can you wait for that PR version to release and give it a shot?

I just pushed a few fixes to the MatchInner reactivity. Can you wait for that PR version to release and give it a shot?

Looks a lot better to me now!
Performance measurement is done without addons again to get a better read on it.
Here is another flame graph:

I just pushed a few fixes to the MatchInner reactivity. Can you wait for that PR version to release and give it a shot?

Looks a lot better to me now! Performance measurement is done without addons again to get a better read on it. Here is another flame graph:

Better or worse than original?

Let me build 2 versions real quick so that devtools won't interfere that much in the measurement
and get back to you in a few minutes!

It's a worse by a one digit millisecond margin compared to 1.34.9.
Didn't attach a flame graph of the production versions since there isn't much to read out of there.
If you want numbers we are talking about 155ms +- 5ms scripting time measured between 1.34.9 and 0f34fbe.
Chance are, this difference comes from other changes made since 1.34.9 and it's on par with what it used to be.
Given that it solves so many bugs it's better than the original. 😄

Perfect! Just need to get tests passing and we’re good. I don’t have anything else to push.

After debugging with a couple of console logs + the React Profile flame graphs, I noticed that route components are rerendered multiple times when hovering links.

To summarize, it seems that when hovering over links the router is rerendering pretty much all Outlet components, hence the React.memo approach to fix the issue.

@tannerlinsley regarding the above mentioned. Is my assumption correct? If so, is this behavior expected (by design) or not really?

@tannerlinsley is there any checklist on what's remaining until this gets merged? Do you think it will be released by the next week?

this needs to be fixed
https://github.com/TanStack/router/pull/1907#issuecomment-2226991746
and of course test need to pass

@tannerlinsley this was probably a mistake so I added that any back:
https://github.com/TanStack/router/pull/1907/commits/2a85a587be896576dc221ceb8a89ec206a823823

Thank you 🕺


Enhancement: re-add ChatGPTbrowser

What features would you like to see added?
Another project that can proxy chatgpt: https://github.com/gngpp/ninja
More details
https://github.com/gngpp/ninja
Which components are impacted by your request?
Endpoints
Pictures
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

you know, I don't want to come across as impolite, and I'm not sure about Danny's perspective on this. However, I don't believe we'll reintroduce a ChatGPT Browser, as all existing ones involve reverse engineering. The only exception I can consider is BingAI. I understand that in some countries, the use of certain LLMs may be restricted, but it's still an activity that is not very lawful

@chokiproai you can use "https://github.com/gngpp/ninja" in the same way PandoraNext was working. You'll have to deploy it yourself tho and use something like that in your .env config:
CHATGPT_TOKEN=user_provided
CHATGPT_MODELS=text-davinci-002-render-sha
CHATGPT_REVERSE_PROXY=http://localhost:7999/backend-api/conversation


Alboroni lz2alerts

Update of Landing Zone alerts to include Traffic Manager, Front Door, App Gateway
Replace this with a brief description of what this Pull Request fixes, changes, etc.
As part of this Pull Request I have

[ x] Read the Contribution Guide and ensured this PR is compliant with the guide
[x ] Checked for duplicate Pull Requests
[x ] Associated it with relevant GitHub Issues or ADO Work Items (Internal Only)
[ ]x Ensured my code/branch is up-to-date with the latest changes in the main branch
[x ] Ensured PR tests are passing
[ ] Updated relevant and associated documentation (e.g. Contribution Guide, Docs etc.)

@Alboroni make sure you replace lines where the managementGroup().name function was used from "policyDefinitionId": "[concat('/providers/Microsoft.Management/managementGroups/',managementGroup().name, to "policyDefinitionId": "[concat('/providers/Microsoft.Management/managementGroups/contoso', in the Deploy-LandingZone-Alerts.json file.
When building the policies.json file the bicep template actually changes that hardcoded value.


Correct the URL used to download JSHint

The URL used to download JSHint from github (https://github.com/bmuschko/gradle-in-action-source/tree/master/chapter11/listing_11_06-07-todo-js-code-quality/ivy-repo/...) in Listing 6-7 in Chapter 11 points the html document for the js file, and not the js file itself.
Because of that running gradle jsHint fails with a lot of warnings since the html is interpreted as js.
I updated jsCodeQuality.gradle to use raw.githubusercontent.com instead, which seems to work as intended.

Good catch. Thanks for the pull request. It has been merged.


API Call for oauth2PermissionGrants

In grantDelegatedPermissionsToApplication.md, the last step before the detections is a POST for the permissions grant against the service principals. As I work through this, I get a 409 conflict which upon additional research seems to indicate that the permissions already exist. It isn't clear to me that because we have done the permissions for mail.read in an earlier step if that is why it is throwing this message or what or if this needs to be a patch instead. Can that be clarified a bit? BTW, this is fantastic work.

Hello @j-stoner ! Thank you very much for the feedback.
Regarding the 409 conflict. The idea is to first add the permissions but then grant it. mmm I wonder if the API to grant is also trying to add the permissions.

Can you share where exactly you get the error message please?

I will do some tests today and improve this section too :)

Yes that section is causing the issue.
I start with the user.read delegated permission with a status of granted and then run the scripts on this page sequentially, starting with List Azure AD Applications and then going to Update Azure AD Application Required Permission and concluding with Grant Delegated Permissions. I have used one or many additional permissions and they are all added correctly including mail.read and directory.real.all.
When I add a trap to the end of this search, the error is
{"error":{"code":"Request_MultipleObjectsWithSameKeyValue","message":"Permission entry already exists.
This makes me think that something about the permissions being added is causing it to think it should be an update and not a create event.
I decided to scramble the section order and ran the scripts in this order; List Azure AD Applications, Grant Delegated Permission, Update Azure AD Application Required Permission. I created a new app, which again only had user.read permission. I did NOT grant admin consent and ran a script that Defined the initial variables of the app and permissions as well as the script under the heading Get Microsoft Graph Service Principal object to retrieve permissions references from it. I then ran script above that was flagged as being a problem.
I got the same error message,
{"error":{"code":"Request_BadRequest","message":"Permission entry already exists
but when I refreshed the Azure UI, the lower section of API Permissions had a heading of Other Permissions Granted For.... and it was populated with those 6 permissions in my above script. So that was progress.
I then ran the permission scripts that were under Update Azure AD Application Required Permissions, ie the first 4 script blocks to assign perms. After a refresh of the UI, things were all good as illustrated in your visuals.

So to make a long story even longer, it looks like while you get a 409 error either way, running the Grant Delegated Permissions script blocks first then the others seem to get you to the desired end state. Not sure if this will vary in other testing but I was trying to solve this for a bit and wanted to see if I was missing something else here.
Hope this is helpful.

I start with the user.read delegated permission with a status of granted and then run the scripts on this page sequentially, starting with List Azure AD Applications and then going to Update Azure AD Application Required Permission and concluding with Grant Delegated Permissions. I have used one or many additional permissions and they are all added correctly including mail.read and directory.real.all.

Does it mean that it worked the first time? You say that it worked with multiple permissions?

I then ran the permission scripts that were under Update Azure AD Application Required Permissions, ie the first 4 script blocks to assign perms. After a refresh of the UI, things were all good as illustrated in your visuals.

which script did you use? this one? https://github.com/Azure/SimuLand/blob/main/2_deploy/aadHybridIdentityADFS/scripts/Update-AzureADAppPermissions.ps1

So to make a long story even longer, it looks like while you get a 409 error either way, running the Grant Delegated Permissions script blocks first then the others seem to get you to the desired end state. Not sure if this will vary in other testing but I was trying to solve this for a bit and wanted to see if I was missing something else here

Would you mind sharing the scripts and code used to get it to work? Maybe if you are using sections from the docs and then the script, can you just point to the script lines in GitHub or sections in the docs? Please

It didn't provide the check with Granted for... when the service principals where run when running the scripts from top to bottom on the page.  This is the result...

Not a problem, just not sure how best to do it.
List Azure AD Applications
$headers = @{“Authorization” = “Bearer $MSGraphAccessToken”} $params = @{   “Method”  = “Get”   “Uri”     = “https://graph.microsoft.com/v1.0/applications”   “Headers” = $headers } $AzADApps = Invoke-RestMethod @params
Define initial variables
$ServicePrincipalId = $AzADAppSp.value[0].id $resourceSpDisplayName = ‘Microsoft Graph’ $PropertyType =  'oauth2PermissionScopes' $permissions = @(‘Mail.ReadWrite’)
Get Microsoft Graph Service Principal object
$headers = @{“Authorization” = “Bearer $MSGraphAccessToken”} $params = @{   "Method" = "Get"   "Uri" = "https://graph.microsoft.com/v1.0/servicePrincipals?$filter=displayName eq '$resourceSpDisplayName'"
"Headers" = $headers
}
$ResourceSvcPrincipal = Invoke-RestMethod @params
if ($ResourceSvcPrincipal.value.Count -ne 1) {
Write-Error "Found $($ResourceSvcPrincipal.value.Count) service principals with displayName '$($resourceSpDisplayName)'"
}
`
Grant permissions with oauth2PermissionGrants API.
`$body = @{
clientId = $ServicePrincipalId
consentType = "AllPrincipals"
principalId = $null
resourceId = $ResourceSvcPrincipal.value[0].id
scope = "$permissions"
startTime = "$((get-date).ToString("yyyy-MM-ddTHH:mm:ss:ffZ"))"
expiryTime = "$((get-date).AddYears(1).ToString("yyyy-MM-ddTHH:mm:ss:ffZ"))"
}
$headers = @{
“Authorization” = “Bearer $MSGraphAccessToken”
"Content-Type" = "application/json"
}
$params = @{
"Method" = "Post"
"Uri" = “https://graph.microsoft.com/v1.0/oauth2PermissionGrants”
"Body" = $body | ConvertTo-Json –Compress
"Headers" = $headers
}
Invoke-RestMethod @params
`
Update Azure AD Application Required Permissions (OAuth2Permissions)
`$Application = $AzADApps.value[0]
$resourceSpDisplayName = ‘Microsoft Graph’
$PropertyType = 'oauth2PermissionScopes'
$ResourceAccessType = 'Scope'
$permissions = @(‘Mail.ReadWrite’)
#Get Microsoft Graph Service Principal object to retrieve permissions from it.
$headers = @{“Authorization” = “Bearer $MSGraphAccessToken”}
$params = @{
"Method" = "Get"
"Uri" = "https://graph.microsoft.com/v1.0/servicePrincipals?$filter=displayName eq '$resourceSpDisplayName'"     "Headers" = $headers } $ResourceResults = Invoke-RestMethod @params $ResourceSvcPrincipal = $ResourceResults.value[0] if ($ResourceResults.value.Count -ne 1) {     Write-Error "Found $($ResourceResults.value.Count) service principals with displayName '$($resourceSpDisplayName)'" }
$ResourceAccessItems = @() Foreach ($AppPermission in $permissions) {     $RoleAssignment = $ResourceSvcPrincipal.$PropertyType | Where-Object { $_.Value -eq $AppPermission }     $ResourceAccessItem = [PSCustomObject]@{         "id" = $RoleAssignment.id         "type" = $ResourceAccessType     }     $ResourceAccessItems += $ResourceAccessItem }
if ($resourceAccess = ($Application.requiredResourceAccess | Where-Object -FilterScript { $_.resourceAppId -eq $ResourceSvcPrincipal.appId })) {     Foreach ($item in $ResourceAccessItems) {         if ($null -eq ($resourceAccess.resourceAccess | Where-Object -FilterScript { $_.type -eq "$ResourceAccessType" -and $_.id -eq $item.id })) {             $Application.requiredResourceAccess[$Application.requiredResourceAccess.resourceAppId.IndexOf($ResourceSvcPrincipal.appId)].resourceAccess += $item          }     } } else {     $RequiredResourceAccess = [PSCustomObject]@{         "resourceAppId" = $ResourceSvcPrincipal.appId         "resourceAccess" = $ResourceAccessItems     }     # Update/Assign application permissions     $Application.requiredResourceAccess += $RequiredResourceAccess }
$AppBody = $Application | Select-Object -Property "id", "appId", "displayName", "identifierUris", "requiredResourceAccess" $headers = @{   “Authorization” = “Bearer $MSGraphAccessToken”   "Content-Type" = "application/json" } $params = @{     "Method" = "Patch"     "Uri" = "https://graph.microsoft.com/v1.0/applications/$($AppBody.id)"     "Body" = $AppBody | ConvertTo-Json -Compress -Depth 99     "Headers" = $headers } $updatedApplication = Invoke-WebRequest @params -usebasicparsing if ($updatedApplication.StatusCode -eq 204) {     return "Required permissions were assigned successfully" }

Not a problem, just not sure how best to do it.
List Azure AD Applications
$headers = @{“Authorization” = “Bearer $MSGraphAccessToken”} $params = @{ “Method” = “Get” “Uri” = “https://graph.microsoft.com/v1.0/applications” “Headers” = $headers } $AzADApps = Invoke-RestMethod @params
Define initial variables
$ServicePrincipalId = $AzADAppSp.value[0].id $resourceSpDisplayName = ‘Microsoft Graph’ $PropertyType = 'oauth2PermissionScopes' $permissions = @(‘Mail.ReadWrite’)
Get Microsoft Graph Service Principal object
$headers = @{“Authorization” = “Bearer $MSGraphAccessToken”} $params = @{ "Method" = "Get" "Uri" = "https://graph.microsoft.com/v1.0/servicePrincipals?$filter=displayName eq '$resourceSpDisplayName'"
"Headers" = $headers
}
$ResourceSvcPrincipal = Invoke-RestMethod @params
if ($ResourceSvcPrincipal.value.Count -ne 1) {
Write-Error "Found $($ResourceSvcPrincipal.value.Count) service principals with displayName '$($resourceSpDisplayName)'"
}
`
Grant permissions with oauth2PermissionGrants API.
`$body = @{
clientId = $ServicePrincipalId
consentType = "AllPrincipals"
principalId = $null
resourceId = $ResourceSvcPrincipal.value[0].id
scope = "$permissions"
startTime = "$((get-date).ToString("yyyy-MM-ddTHH:mm:ss:ffZ"))"
expiryTime = "$((get-date).AddYears(1).ToString("yyyy-MM-ddTHH:mm:ss:ffZ"))"
}
$headers = @{
“Authorization” = “Bearer $MSGraphAccessToken”
"Content-Type" = "application/json"
}
$params = @{
"Method" = "Post"
"Uri" = “https://graph.microsoft.com/v1.0/oauth2PermissionGrants”
"Body" = $body | ConvertTo-Json –Compress
"Headers" = $headers
}
Invoke-RestMethod @params
`
Update Azure AD Application Required Permissions (OAuth2Permissions)
`$Application = $AzADApps.value[0]
$resourceSpDisplayName = ‘Microsoft Graph’
$PropertyType = 'oauth2PermissionScopes'
$ResourceAccessType = 'Scope'
$permissions = @(‘Mail.ReadWrite’)
#Get Microsoft Graph Service Principal object to retrieve permissions from it.
$headers = @{“Authorization” = “Bearer $MSGraphAccessToken”}
$params = @{
"Method" = "Get"
"Uri" = "[https://graph.microsoft.com/v1.0/servicePrincipals?$filter=displayName](https://graph.microsoft.com/v1.0/servicePrincipals?%60$filter=displayName) eq '$resourceSpDisplayName'" "Headers" = $headers } $ResourceResults = Invoke-RestMethod @params $ResourceSvcPrincipal = $ResourceResults.value[0] if ($ResourceResults.value.Count -ne 1) { Write-Error "Found $($ResourceResults.value.Count) service principals with displayName '$($resourceSpDisplayName)'" }
$ResourceAccessItems = @() Foreach ($AppPermission in $permissions) { $RoleAssignment = $ResourceSvcPrincipal.$PropertyType | Where-Object { $_.Value -eq $AppPermission } $ResourceAccessItem = [PSCustomObject]@{ "id" = $RoleAssignment.id "type" = $ResourceAccessType } $ResourceAccessItems += $ResourceAccessItem }
if ($resourceAccess = ($Application.requiredResourceAccess | Where-Object -FilterScript { $_.resourceAppId -eq $ResourceSvcPrincipal.appId })) { Foreach ($item in $ResourceAccessItems) { if ($null -eq ($resourceAccess.resourceAccess | Where-Object -FilterScript { $_.type -eq "$ResourceAccessType" -and $_.id -eq $item.id })) { $Application.requiredResourceAccess[$Application.requiredResourceAccess.resourceAppId.IndexOf($ResourceSvcPrincipal.appId)].resourceAccess += $item } } } else { $RequiredResourceAccess = [PSCustomObject]@{ "resourceAppId" = $ResourceSvcPrincipal.appId "resourceAccess" = $ResourceAccessItems } # Update/Assign application permissions $Application.requiredResourceAccess += $RequiredResourceAccess }
$AppBody = $Application | Select-Object -Property "id", "appId", "displayName", "identifierUris", "requiredResourceAccess" $headers = @{ “Authorization” = “Bearer $MSGraphAccessToken” "Content-Type" = "application/json" } $params = @{ "Method" = "Patch" "Uri" = "https://graph.microsoft.com/v1.0/applications/$($AppBody.id)" "Body" = $AppBody | ConvertTo-Json -Compress -Depth 99 "Headers" = $headers } $updatedApplication = Invoke-WebRequest @params -usebasicparsing if ($updatedApplication.StatusCode -eq 204) { return "Required permissions were assigned successfully" }

Niceee!!! So all that code worked to get the Granted check mark right? @j-stoner

Yeah, it was doing the service principal first then permissions that seemed to work...both threw 409 which is odd....

Ohh so even when getting it work, it stills returns a 409 response code .Interesting. I am testing it today.

Ok I figured it our after some testing. The issue that I had was the when I register an AD application for this lab, I was adding the User.Read delegated permissions and granting admin consent to it. Somehow, the Grant API was not liking me to grant another permissions when there was already another one granted.
I updated the instructions that after deploying the Hybrid environment for the first lab, an Azure AD app needs to be registered and added a delegated permissions User.Read, but it does not need to be granted admin consent.
After that, I ran through all the steps to update the application permissions and grant Mail.ReadWrite permissions and it worked fine.  Thank you very much for all the details. It was very helpful to figure it out :)
I appreciate you taking the time @j-stoner to open the issue and provide details. Thank you!