Refactor config attributes loading

Extends the validation of the mandatory_dict_keys in config by
checking that they also have a value. There is no point in having checks
for mandatory keys if they are empty (this would mean that they are not
really mandatory)
Extracts _parse_config method to utilize for loading configs
Moves the self._config check from _verify_dict method to beginning
of initialization. If the main config file is wrong, there's no need
proceeding any more
Extracts plugin loading to _load_plugins method
Fixes #184 by setting empty list as default value for plugin configs
in case a specific plugin config is unset thus allowing MICRO_SERVICES
plugin to be empty (instead of deleting the key)
Adds test cases for invalid internal attributes and empty
MICRO_SERVICES plugin
Adds test case for invalid conf although since the exception thrown is
generic for other config cases as well, the only way to verify this
is by checking the coverage (subclassing the config errors is probably a
better alternative)

All Submissions:

[X] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[X] Have you added an explanation of what problem you are trying to solve with this PR?
[X] Have you added information on what your changes do and why you chose this as your solution?
[X] Have you written new tests for your changes?
[X] Does your submission pass tests?
[X] This project follows PEP8 style guide.

Extends the validation of the mandatory_dict_keys...
Extracts _parse_config...
Moves the self._config check...
Extracts plugin loading...
Fixes #184 ...
Adds test cases for invalid internal attributes...
Adds test case for invalid conf...

There are 7 bullets describing changes, but only two commits. Can we balance this in the future?

If the main config file is wrong, there's no need proceeding any more

If there different config files, their checks are independent. I would like to know all the errors I could possibly know with a single run, fix all I can fix, and retry.

subclassing the config errors is probably a better alternative

Yes, we should do this. Are there any other alternatives?

Extracts _parse_config method to utilize for loading configs
Extracts plugin loading to _load_plugins method

This is good; but the interesting part is why. Why do these need to be extracted? How is it better having them separate?
Ideally, this should be part of the commit message that introduces the respective change.

There are 7 bullets describing changes, but only two commits. Can we balance this in the future?
Sure, I can rebase and split them to more commits

If the main config file is wrong, there's no need proceeding any more
If there different config files, their checks are independent. I would like to know all the errors I could possibly know with a single run, fix all I can fix, and retry.

The original behavior changed didn't change since my purpose was to refactor the code and not change the behavior. Every stop and every error is in the same order as it was before. I think that if we wish to change the behavior to that which you describe, it should be in a different PR not in the same PR as the refactoring.

subclassing the config errors is probably a better alternative

Yes, we should do this. Are there any other alternatives?
Sure there are alternatives to subclassing (i.e carrying the erroneous config as a class variable could be one alternative) but I'm not sure we need this.

This is good; but the interesting part is why. Why do these need to be extracted? How is it better having them separate?

Initially this PR was about #184 . Reading through the code I realized that the __init__ method was too big (for my taste) and also had some deeply nesting (which resulted in cyclomatic complexity 12). So I decided to refactor this to reduce the complexity and at the same time make the constructor less verbose hiding the details in other methods. I think it is kind of obvious about why this extraction is useful. Otherwise the _verify_dict and the load_yaml and load_dict methods would be inside the constructor as well.

If the main config file is wrong, there's no need proceeding any more
If there different config files, their checks are independent. I would like to know all the errors I could possibly know with a single run, fix all I can fix, and retry.

The original behavior didn't change since my purpose was to refactor the code and not change the behavior. Every stop and every error is in the same order as it was before. I think that if we wish to change the behavior to that which you describe, it should be in a different PR not in the same PR as the refactoring.

We do not change things just because they can be changed. Refactoring is not about moving things around. The refactor should have a goal which should be stated and it should answer how it makes things "better". In this process it is definitely desirable to group things differently (prepare for the changes) and then change how they behave (make things better), to make the transition clear and understandable (it would be also be reflected by the commits.) But still, the goal is to change things to make them better.
How far we want to go with that is what we discuss here. I'm all in for pushing that a lot. If you feel we should restrict this and have separate PRs for related aspects, let's do that.

subclassing the config errors is probably a better alternative

Yes, we should do this. Are there any other alternatives?

Sure there are alternatives to subclassing (i.e carrying the erroneous config as a class variable could be one alternative) but I'm not sure we need this.

We don't need what? Do you mean we do not need alternatives?
How does "carrying the erroneous config as a class variable" help?
Just to remind us, what we try to solve here is what you initially wrote: how can we know that an exception we expect came from a specific handler?

This is good; but the interesting part is why. Why do these need to be extracted? How is it better having them separate?

Initially this PR was about #184 . Reading through the code I realized that the __init__ method was too big (for my taste) and also had some deeply nesting (which resulted in cyclomatic complexity 12). So I decided to refactor this to reduce the complexity and at the same time make the constructor less verbose hiding the details in other methods. I think it is kind of obvious about why this extraction is useful. Otherwise the _verify_dict and the load_yaml and load_dict methods would be inside the constructor as well.

It is not obvious to everyone, as it was not to the original author. Additionally, it may not be obvious at all, when three years later all the surrounding code has changed. Having this as part of the history is very helpful when someone will need to go through it and understand. This particular case may not be very important, but we should still do this to push our thinking in that direction.

If the main config file is wrong, there's no need proceeding any more
If there different config files, their checks are independent. I would like to know all the errors I could possibly know with a single run, fix all I can fix, and retry.

The original behavior didn't change since my purpose was to refactor the code and not change the behavior. Every stop and every error is in the same order as it was before. I think that if we wish to change the behavior to that which you describe, it should be in a different PR not in the same PR as the refactoring.

We do not change things just because they can be changed. Refactoring is not about moving things around. The refactor should have a goal which should be stated and it should answer how it makes things "better". In this process it is definitely desirable to group things differently (prepare for the changes) and then change how they behave (make things better), to make the transition clear and understandable (it would be also be reflected by the commits.) But still, the goal is to change things to make them better.

As I said, the point of this refactoring (which came as a side effect of #184) was to improve readability by reducing the nested for loops, reducing complexity (same way), and extracting operations with a mind of reusability (without pushing reusability too much to end up in coupling). There doesn't have to be always a behavioral change or feature to refactor although the real value of the refactoring is to restructure the code in a way that is easier to maintain, extend and change. So with that mindset, I believe that we should change things because they can be changed to provide the aforementioned things.

How far we want to go with that is what we discuss here. I'm all in for pushing that a lot. If you feel we should restrict this and have separate PRs for related aspects, let's do that.

I really believe this PR discussion has gone way out of track and the PR lost it's value.

subclassing the config errors is probably a better alternative

Yes, we should do this. Are there any other alternatives?

Sure there are alternatives to subclassing (i.e carrying the erroneous config as a class variable could be one alternative) but I'm not sure we need this.

We don't need what? Do you mean we do not need alternatives?
How does "carrying the erroneous config as a class variable" help?
Just to remind us, what we try to solve here is what you initially wrote: how can we know that an exception we expect came from a specific handler?

Carrying the erroneous config would help to discern which config was erroneous not trace the line/caller the exception came from. I wrote what I wrote because this exception class was used everywhere and the only way to check the correct one was thrown was to check the error message which I consider wrong. If for some reason (what would that be?) one wants to trace the function caller that resulted in the exception, they could as well use the traceback module. It's like trying to trace the exit point of a multiple-exit-point function

This is good; but the interesting part is why. Why do these need to be extracted? How is it better having them separate?

Initially this PR was about #184 . Reading through the code I realized that the __init__ method was too big (for my taste) and also had some deeply nesting (which resulted in cyclomatic complexity 12). So I decided to refactor this to reduce the complexity and at the same time make the constructor less verbose hiding the details in other methods. I think it is kind of obvious about why this extraction is useful. Otherwise the _verify_dict and the load_yaml and load_dict methods would be inside the constructor as well.

It is not obvious to everyone, as it was not to the original author. Additionally, it may not be obvious at all, when three years later all the surrounding code has changed. Having this as part of the history is very helpful when someone will need to go through it and understand. This particular case may not be very important, but we should still do this to push our thinking in that direction.

I'm still not sure what explanation to write to justify an extraction to a method. Splitting functionality would need explanation (if for example the plugins loading was split instead of being in a for loop), but I consider this a really minor refactoring to explain and justify the extraction and the rewrite of a 4 level nested for loop

I prefer to close this PR. This started as a simple and harmless imho refactoring to improve readability and complexity on the track of solving a minor issue and ended up discussing about behavioral changes, functional paradigms, readability and exception subclassing alternatives. What started as a simple change request, ended up in the longest PR discussion for this repo. I've really lost sight of what needs to be done here. So it's better to close this and handle anything that is needed after a corresponding issue is opened that would explain what needs to be done.

How far we want to go with that is what we discuss here. I'm all in for pushing that a lot. If you feel we should restrict this and have separate PRs for related aspects, let's do that.

I really believe this PR discussion has gone way out of track and the PR lost it's value.

The value is still here, both in the code and the discussion. And I actually believe that you've probably already learnt new things from these discussions. I did, others reading this did. There is no reason to dismiss this.

subclassing the config errors is probably a better alternative

Yes, we should do this. Are there any other alternatives?

Sure there are alternatives to subclassing (i.e carrying the erroneous config as a class variable could be one alternative) but I'm not sure we need this.

We don't need what? Do you mean we do not need alternatives?
How does "carrying the erroneous config as a class variable" help?
Just to remind us, what we try to solve here is what you initially wrote: how can we know that an exception we expect came from a specific handler?

Carrying the erroneous config would help to discern which config was erroneous not trace the line/caller the exception came from.

Yes, we do not care about the line; we care about the handler in the sense you describe - not the handler as a caller, but the operation it was doing that will help us understand what went wrong.

I wrote what I wrote because this exception class was used everywhere and the only way to check the correct one was thrown was to check the error message which I consider wrong.

Right! We should not check the error message; and even if we did, it could be the same, so this is a no-go.
So, this topic is a super interesting topic, and I believe it is in the heart of complexity in software. It is also why monads are so useful (at least, for my understanding of monads). What we see here is the need to propagate the state of the operation to produce a meaningful error message. The same thing is needed in either direction of the flow. This propagation of state is useful both to stop and analyse the program, but also to progress the program and to make decisions.
There are simple ways to make this happen (save the information in some context), and this is what we will do for a start. But modelling the generic mechanism to get that information without having to redo the tedious plumbing is very interesting, and leads to a different paradigm; it is not about types, purity, or what-not, it is about information accumulation and decisions.

This is good; but the interesting part is why. Why do these need to be extracted? How is it better having them separate?

Initially this PR was about #184 . Reading through the code I realized that the __init__ method was too big (for my taste) and also had some deeply nesting (which resulted in cyclomatic complexity 12). So I decided to refactor this to reduce the complexity and at the same time make the constructor less verbose hiding the details in other methods. I think it is kind of obvious about why this extraction is useful. Otherwise the _verify_dict and the load_yaml and load_dict methods would be inside the constructor as well.

It is not obvious to everyone, as it was not to the original author. Additionally, it may not be obvious at all, when three years later all the surrounding code has changed. Having this as part of the history is very helpful when someone will need to go through it and understand. This particular case may not be very important, but we should still do this to push our thinking in that direction.

I'm still not sure what explanation to write to justify an extraction to a method. Splitting functionality would need explanation (if for example the plugins loading was split instead of being in a for loop), but I consider this a really minor refactoring to explain and justify the extraction and the rewrite of a 4 level nested for loop

You've already written that you though it was verbose, too big and complex. These are good reasons, but you're basing them on "taste" or some number that is supposed to indicate complexity, but you know that there is something bothering you and it is not the number of lines, neither the CC, because you have seen other code that may be longer or more complex, but it still felt fine.
The hard question is what makes anyone judge something as verbose, big or complex. In a previous PR you wrote something related and very important, and you wrote a similar thing elsewhere in this PR. Let me quote:

Now there are 3 functions: one for the creation of the metadata objects, one for the writing the metadata objects to file(s), and one for composing those two to a final result

So, what drives the split is exactly what you write: you want to separate concerns, and then you want to compose them to achieve a bigger effect. There is more than composition, but separating concerns and composing them is a first step that makes this feeling that is bothering us and that we attributed to taste, an actual guide to making such decisions. The way you separate concerns is by figuring out the level of detail that describe an operation. Finer details are encapsulated and split into separate functions, so that the function is concerned (not only with a single aspect, but) with details of the same abstraction level.
The next step is orchestration: every function is an orchestrator of data and effects. Values are derived by transforming data, where transformations are compositions of functions. Business results are derived by chaining effects that are built on top of those compositions. Orchestration provides a higher level way to think about and deal with the concerns of business requirements. It allows us to understand the abstraction details of business operations (which are implemented as steps of function compositions).

I prefer to close this PR. This started as a simple and harmless imho refactoring to improve readability and complexity on the track of solving a minor issue and ended up discussing about behavioral changes, functional paradigms, readability and exception subclassing alternatives.

This is a discussion. We explore what can be done better. It is the only way to design things, and I know you are capable of doing that. (I am also pretty sure you want to do that.)
All the things you describe above, are rooted to deeper questions, sometimes annoyingly abstract, and these are the questions I try to bring out. I dedicate time to this, because I know you can contribute to these questions and I know that many of these are things that have been bothering you, too.

What started as a simple change request, ended up in the longest PR discussion for this repo.

The length of the PR discussion is irrelevant.

I've really lost sight of what needs to be done here. So it's better to close this and handle anything that is needed after a corresponding issue is opened that would explain what needs to be done.

What needs to be done is what we will decide by discussing the different tradeoffs of the alternative choices we have. The reason to have multiple comments and not a big one, is so that we can conclude for each point separately.
Are you interested in doing this with me?

https://github.com/IdentityPython/SATOSA/pull/251#issuecomment-510688196 I totally agree

I prefer to close this PR. This started as a simple and harmless imho refactoring to improve readability and complexity on the track of solving a minor issue and ended up discussing about behavioral changes, functional paradigms, readability and exception subclassing alternatives.

This is a discussion. We explore what can be done better. It is the only way to design things, and I know you are capable of doing that. (I am also pretty sure you want to do that.)
All the things you describe above, are rooted to deeper questions, sometimes annoyingly abstract, and these are the questions I try to bring out. I dedicate time to this, because I know you can contribute to these questions and I know that many of these are things that have been bothering you, too.

What started as a simple change request, ended up in the longest PR discussion for this repo.

The length of the PR discussion is irrelevant.

I've really lost sight of what needs to be done here. So it's better to close this and handle anything that is needed after a corresponding issue is opened that would explain what needs to be done.

What needs to be done is what we will decide by discussing the different tradeoffs of the alternative choices we have. The reason to have multiple comments and not a big one, is so that we can conclude for each point separately.
Are you interested in doing this with me?

I'm more than interested in trying to discover possible answers to these questions. These questions are the real beauty of programming and the journey to find these answers is what is worth the try, not trying to get a working result.
My point was that this PR started as a simple refactoring to initiate a possible future bigger refactoring/rewrite of things in order to improve maintainability, extensibility and plugability (someone could say it is the same with extensibility). So I think that the discussion should discuss if this refactoring is paving the aforementioned road or not, not try to define every detail of that road here in comments. Designing the details of that road should (imho) be analyzed and discussed in a separate discussion so as to have the bigger picture in mind instead of discussing every detail (because details are a part of the bigger picture). In this case, since this refactoring raised concerns about the way the configuration is read, validated and handled, a separate issue should be opened to discuss how we want these things to change for a new PR to follow. Discussing these in this PR is losing the bigger picture (since we're talking in specific snippets of code) and in a way is denoting (at least in my mind) that this PR is following that path.

I'm more than interested in trying to discover possible answers to these questions.

cool; so I'm reopening this 🙂

My point was that this PR started as a simple refactoring to initiate a possible future bigger refactoring/rewrite of things
[we] should discuss if this refactoring is paving the aforementioned road or not

yes and no. For the local context of the mechanisms that handle the configurations, yes; it is fixing parts of the existing way of doing things, which is good. But, in the long run, the config format itself should be revisited. In other words, this PR is accepting that the current data source is correct; while I'm saying that to make this right we should start by modelling the data we get first.
I haven't mentioned that in the rest of the discussion, and it's not what we should focus on now. That's because there are things that will be the same regardless of the config format - it's not like everything needs to be written from scratch. These are the things I'm trying to keep this discussion focused on. Not the details of the handling and parsing, but the structure of the logic that we need in place.
These discussions are not tied to the config itself, but apply elsewhere too.
To be more precise, the questions about the difference of the error messages, who checks for the error, keeping a straight code-path, how to make clear what happened, and getting errors at once about the whole input, are all things that should be rewritten with a data-first approach. (We will see what this means in practice.)

Designing the details of that road should (imho) be analyzed and discussed in a separate discussion so as to have the bigger picture in mind instead of discussing every detail

again, yes and no. Discussions need to start somewhere. Once we figure what we need to change or how, we can write a clean description of what we discussed and to what we arrived. This cannot be done the other way around. Discussion needs to take place first. Then we can summarize and state the tradeoffs and the decision. Then make changes.
If we knew what should be done there would be no need to discuss it. So, I think it is good to have those discussions. Some will lead nowhere, some will lead to immediate changes, and others will unwind until we have a better understanding - at which point we can summarize in a separate email, issue or PR.

 I just want to say that it happened, Should I leave it there? I dunno, but I think that my son shouldn't read it, he should never code in this way, but it exists and it works and I'll never use debugger there. Sorry, I just want to say that things happen and somethime we should let exists :')

Yesterday I coded this incredible (pythonic?!?) bunch of things:
https://github.com/peppelinux/pyLDAP/blob/master/client.py#L46
Today when I read it my thought has come to you, here... I just want to say that it happened, Should I leave it there? I dunno, but I think that my son shouldn't read it, he should never code in this way, but it exists and it works and I'll never use debugger there. Sorry, I just want to say that things happen and somethime we should let exists :')

if it makes you feel any better, that line cannot be written as a one-liner with map/filter. but even if you choose to do it in 2 lines, the result is still bad. so it would be better to refactor it unless you want to explain this to your son :stuck_out_tongue:

Great! I Just want to explain that everything could be refactored but the ideas are the most important things. Thank you for your suggestion, I'll back to that!


Bug Report: Change UI of login page

Reproduction steps
The current UI of login page is not looking very good.
Expected behavior
UI is expected to look neat and professional
Proposed UI:
Similar to sign up page with a different image

Actual Behavior
Current UI:

👀 Have you spent some time to check if this issue has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Hi, I would like to work on this issue. can you assign me this?

Hi, I would like to work on this issue. can you assign me this?

Hi, I would like to work on this issue. can you assign me this?


KeyError: "The name 'import/final_result' refers to an Operation not in the graph."

Traceback (most recent call last):
File "D:/xcr/human-action-classification-master/human-action-classification-master/run_webcam.py", line 65, in
pose_class = label_img.classify(image)
File "D:\xcr\human-action-classification-master\human-action-classification-master\scripts\label_image.py", line 99, in classify
output_operation = graph.get_operation_by_name(output_name)
File "C:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3606, in get_operation_by_name
return self.as_graph_element(name, allow_tensor=False, allow_operation=True)
File "C:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3478, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "C:\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3538, in _as_graph_element_locked
"graph." % repr(name))
KeyError: "The name 'import/final_result' refers to an Operation not in the graph."

@shaowujie,
In order to reproduce the issue reported here, could you please provide the TensorFlow version, the complete code and the dataset you are using. Thanks!

Hi there,
I have generated the same error:
2022-01-15 13:02:21.671933: W tensorflow/core/framework/op_def_util.cc:355] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization().
Traceback (most recent call last):
File "run_image.py", line 51, in
scene_class = label_img_scene.classify(args.image)
File "C:\Users\libby\Documents\Python\new-human-action-classification-g\human-action-classification-g\scripts\label_image_scene.py", line 87, in classify
output_operation = graph.get_operation_by_name(output_name);
File "C:\Users\libby\Anaconda3\envs\action-classification\lib\site-packages\tensorflow\python\framework\ops.py", line 3606, in get_operation_by_name
return self.as_graph_element(name, allow_tensor=False, allow_operation=True)
File "C:\Users\libby\Anaconda3\envs\action-classification\lib\site-packages\tensorflow\python\framework\ops.py", line 3478, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "C:\Users\libby\Anaconda3\envs\action-classification\lib\site-packages\tensorflow\python\framework\ops.py", line 3538, in _as_graph_element_locked
"graph." % repr(name))
KeyError: "The name 'import/final_result' refers to an Operation not in the graph."
I'm on Tensorflow 1.13.1


Juno Compatibility Test

We perform the following tests to verify Cloudmesh running on OpenStack Juno:

vm creation
vm deletion
vm list
flavor list
image list
ip allocation
ip deletion
keypair add
keypair delete
keypair list

Nosetest is preferred to provide test results.

I had a simple test. I've tried cloud activation, vm start, vm delete... it
all work!  But there are some places in cloudmesh need to be changed...
2015-02-09 15:45 GMT-05:00 Hyungro Lee<EMAIL_ADDRESS>
Assigned #188 https://github.com/cloudmesh/cloudmesh/issues/188 to
@markxiao https://github.com/markxiao.
—
Reply to this email directly or view it on GitHub
https://github.com/cloudmesh/cloudmesh/issues/188#event-232757287.

What are they? Can you share which place we need to update?

I had some problem reading the yaml file, I didn't look into it, it might
be my operational error. I added following cloud info in the cloudmesh.yaml
after cloud india:
india-juno:
cm_heading: India OpenStack, Juno
cm_host: india.futuregrid.org
cm_label: ios_juno
cm_service_url_type: publicURL
cm_type: openstack
cm_type_version: juno
credentials:
OS_AUTH_URL: https://i5r.idp.iu.futuregrid.org:5000/v2.0
OS_CACERT:
/home/marktest/.cloudmesh/clouds/india/juno/cacert.pem
OS_PASSWORD: ***
OS_TENANT_NAME: ***
OS_USERNAME: ***
OS_VERSION: juno
default:
flavor: None
image: None
Then restart the server, cloud india-juno is in the cloud list but the
information shown looks like some default value such as "123456". I
manually changed it and I can activate the cloud successfully.
Besides this, CLI can't show vm addresses (GUI works fine) due to the path
to data in the database is different between havana and juno.
I will let you know if I find more.
2015-02-09 16:29 GMT-05:00 Hyungro Lee<EMAIL_ADDRESS>
What are they? Can you share which place we need to update?
—
Reply to this email directly or view it on GitHub
https://github.com/cloudmesh/cloudmesh/issues/188#issuecomment-73593967.


HOL 1 - Separate the node and java steps in the readme

Were we going to clone the Java steps into Node & Dotnet and make a mental note to keep them in sync going forward? Or something else? While i'd hate updating 3 locations, I'm afraid gutting the readmes for Node & Dotnet, and putting in a redirect hyperlink  to Java may be insufficient.
Thoughts on approach?

Should also focus the readmes for HOL1 ex. dotnet developers shouldn't mess with updating Nodejs or installing Eclipse

I moved HOL 1 to the shared directory, but there are a couple of steps that are only for Java developers (and they are marked that way).  I'm leaving the NodeJS install because they'll need that to install the xplat CLI.

@rsponholtz Are you OK if i move the HOL1 from the shared folder back to each of the three track's HOL folders? While a bit more overhead to keep in sync, we had issues with the Shared setup this week in Munich. It makes attendees bounce around searching for files across different areas of the repo, vs. it just being there.  Can also strip out the non-track specific instructions (removing Java steps for the Dotnet track, etc)

Will move from shared into 3 separate tracks


[CI] Mark a stable image from release candidate

Use case

Describe the solution you'd like

Describe alternatives you've considered

Additional context

It's okay for the CI to only generate .-rc tag
and after manual verification, a developer to add a new tag to ghcr.io without the rc. For example
1.3.20-rc is generate by CI
After tests, someone adds a new tag 1.3.20 to the same image. Our deployment script will use 1.3.20 as the tag for Timeplus Cloud update.
CC @leo-cai-timeplus ,  @qijun-niu-timeplus  @jhao0117  @chenziliang  @gangtao @zliang-min


Opensearch/Elasticsearch support #542

Closes issue #542.
Changes included

An OpensearchVectorClient client that encapsulates all the config and logic for using an opensearch index as a vector store. The reason this is broken up into its own client is mainly for consistency with the design of the existing VectorStores. That is, all the existing VectorStore take in a single "client" object, so breaking out the client into a separate class made more sense here.

Initially I wanted to use langchain's VectorStore for Opensearch as the client but the api exposed by that one didn't gel well, so it made more sense to create a new client abstraction.

A VectorStore for Opensearch that uses said client

Lucene vector indices don't support a high enough dimension to be worth using, and Elasticsearch does not support other vector indices. Although I could add an Elasticsearch VectorStore in a future PR for consistency's sake.)

A GPTVectorStoreIndex built ontop of said vector store.
A BaseReader for Elasticsearch/Opensearch

Works with either since the reader just uses the standard text indices and doesn't rely on any engine-specific features.

An example--also served as my integration test 😅 as I didn't see a place in the test suite for that type of testing.

huge change! thanks @jaylmiller, will review soon

huge change! thanks @jaylmiller, will review soon

It's not actually that much code, the ipynb file just makes it look huge 😅

huge change! thanks @jaylmiller, will review soon

It's not actually that much code, the ipynb file just makes it look huge 😅

oh i meant huge in that i'm really excited about it, not in terms of lines of code :) 😂

Done 🌞

sweet! quick heads up i'm looking to land this tomorrow (cut a release wednesday morning)

I can't wait for it!

out of curiosity, how did you get opensearch setup? i'm trying to test the notebook and i keep getting ssl errors

nvm figured it out! what's your twitter handle? i'll release this tomorrow morning

nvm figured it out! what's your twitter handle? i'll release this tomorrow morning

im not on twitter haha


Add Portuga Manga

Checklist:

[x] Updated extVersionCode value in build.gradle for individual extensions
[x] Updated overrideVersionCode or baseVersionCode as needed for all multisrc extensions
[ ] Referenced all related issues in the PR body (e.g. "Closes #xyz")
[x] Added the isNsfw = true flag in build.gradle when appropriate
[x] Have not changed source names
[ ] Have explicitly kept the id if a source's name or language were changed
[ ] Have tested the modifications by compiling and running the extension through Android Studio

main was force pushed, rebase and open a new pr. Sorry for the trouble!!

Hey, to migrate your change, on your branch do
git remote update
git reset --hard upstream/main
git cherry-pick 83c04a8^..4a61d79
git push -f


Metrics seems to fail when not defining a Hawkular Endpoint.

When not defining a Hawkular Endpoint hostname metric collection fails, if should fallback to the providers hostname.

@miq-bot add_label providers/containers
cc @simon3z

@miq-bot assign yaacov

Fixed by: https://github.com/ManageIQ/manageiq/pull/9025


RN 0.43 upgrade error ReactComponentWithPureRenderMixin

Description
error: bundling: UnableToResolveError: Unable to resolve module react/lib/ReactComponentWithPureRenderMixin from C:\project\node_modules\react-addons-pure-render-mixin\index.js: Module does not exist in the module map or in these directories:
C:\project\node_modules\react\lib
Reproduction Steps and Sample Code
After Upgrade from 0.42 to 0.43 we receive the following error. Requested file does not exist on node_modules.
We have clear node_modules and run "npm install"
We are using "react": "16.0.0-alpha.6"
Additional Information

React Native version: 0.43
Platform: Android
Development Operating System: Windows
Dev tools: -

Here is the excerpt from react documentation

The PureRenderMixin mixin predates React.PureComponent. This reference doc is provided for legacy purposes, and you should consider using React.PureComponent instead.

Similar problem here

There are multiple blog posts on react won't be supporting mixins anymore. Also above excerpt from react documentation recommends PureComponent instead of PureRenderMixin. I did the same.

https://medium.com/@dan_abramov/mixins-are-dead-long-live-higher-order-components-94a0d2f9e750

It seems that "react-native-router-flux" cause the problem to me because it has dependency with "react-addons-pure-render-mixin"

I've integrated ReactNavigation today, and I'm facing with this issue.
Any solutions for this please, maybe downgrade @react version or @react-navigation version?
 "dependencies": {     "react": "^15.4.2",     "react-native": "0.43.3",     "react-navigation": "^1.0.0-beta.7"   },   "devDependencies": {     "babel-jest": "19.0.0",     "babel-preset-react-native": "1.9.1",     "jest": "19.0.2",     "react-test-renderer": "^16.0.0-alpha.8"   },

same problem here

In case you are using react-navigation, make sure, you are using the master branch.
npm i --save react-community/react-navigation#master

With yarn, you have to add the master as well:
yarn add react-community/react-navigation#master

This is not an issue of the react-native project as those methods are deprecated for some time. The problem is caused by external libs/deps you're using...

When you upgrade,it will upgrade react version too.now it is using react version 16.alpha. in 16 there is no pure-render-mixin.so you need to install it through npm.or you have to downgrade react version into 15.x.
Install these will fix the issue.
"react-addons-pure-render-mixin": "^15.5.0",
"react-static-container": "^1.0.1",

@Chathula This helps one step ahead, thx :)
And continues then with
bundling: UnableToResolveError: Unable to resolve module react/lib/ReactComponentTreeHook from "/node_modules/react-native/Libraries/Performance/Systrace.js"

@itinance, can you post your package.json here?
What did u do? Downgrade or install those two packages?

I'm using:
"react": "16.0.0-alpha.3",
"react-native": "0.43.3",

And it works for me, but when trying to increase to react alpha.6 or 7 it does not work.

Oh sorry @Chathula, didn't get the notification. Thank you for reaching out.
This was the most recent version of package.json where we tried to bring up RN 0.43 without any success 10 days ago. Of course there happened further version updates in the meanwhile that are not recorded here because we stopped the upgrade and remained with 0.42 / 0.40 (two separate apps)
    "dependencies": {
        "apsl-react-native-button": "^3.0.2",
        "babel-plugin-idx": "^1.1.0",
        "bugsnag-react-native": "^2.2.0",
        "color": "^1.0.3",
        "color-convert": "^1.9.0",
        "d3-format": "https://github.com/d3/d3-format",
        "deepcopy": "^0.6.3",
        "idx": "^1.1.0",
        "immutable": "^3.8.1",
        "keymirror": "^0.1.1",
        "lodash.isequal": "^4.5.0",
        "lodash.trim": "^4.5.1",
        "lodash.uniqby": "^4.7.0",
        "lodash.uniqwith": "^4.5.0",
        "moment": "^2.17.1",
        "react": "16.0.0.-alpha.9",
        "react-addons-shallow-compare": "^15.4.2",
        "react-addons-pure-render-mixin": "^15.5.0",
        "react-static-container": "^1.0.1",
        "react-native": "0.43.2",
        "react-native-animatable": "^1.1.0",
        "react-native-busy-indicator": "^1.0.9",
        "react-native-checkbox": "^1.1.0",
        "react-native-collapsible": "^0.8.0",
        "react-native-datepicker": "^1.4.3",
        "react-native-device-info": "^0.9.9",
        "react-native-drawer": "^2.3.0",
        "react-native-extra-dimensions-android": "^0.20.1",
        "react-native-icon-badge": "^1.1.1",
        "react-native-ikon": "^0.1.2",
        "react-native-item-cell": "^1.6.0",
        "react-native-modalbox": "latest",
        "react-native-navbar": "^1.6.0",
        "react-native-parsed-text": "^0.0.16",
        "react-native-picker": "^4.0.18",
        "react-native-router-flux": "itinance/react-native-router-flux#c3bdac5a4cb850c049524385ffe84826ea77fb90",
        "react-native-side-menu": "^0.20.1",
        "react-native-simple-radio-button": "^2.5.0-alpha.1",
        "react-native-sqlite-storage": "^3.2.2",
        "react-native-swiper": "^1.5.4",
        "react-native-switch": "^1.0.5",
        "react-native-uuid-generator": "^2.0.0",
        "react-native-vector-icons": "^4.0.0",
        "react-native-version-number": "^0.1.1",
        "react-redux": "^5.0.2",
        "redux": "^3.6.0",
        "redux-form": "^6.5.0",
        "redux-logger": "^2.8.1",
        "redux-saga": "^0.14.3",
        "redux-thunk": "^2.2.0",
        "reselect": "^2.5.4",
        "rn-viewpager": "^1.1.4",
        "slim-formatter": "^1.0.6",
        "sprintf-js": "^1.0.3",
        "standard-http-error": "^2.0.0",
        "style-equal": "^1.0.0"
    },
    "devDependencies": {
        "babel-jest": "18.0.0",
        "babel-preset-react-native": "1.9.1",
        "jest": "18.1.0",
        "jest-react-native": "^18.0.0",
        "react-test-renderer": "15.4.2"
    },

@itinance have you tried with "react": "16.0.0.-alpha.3" instead of "react": "16.0.0.-alpha.9" ?

Recent experience suggests that until these things hit release, it’s probably best to stick with the exact versions specified as peerDependencies by react-native. Notice in package.json that react-native itself specifies exact versions, not ranges. Let your project dependency be "react": "16.0.0-alpha.$foo" according to the day’s react-native`, not "^16.0.0-***".
I’m guessing that React (being alpha software) has a very unstable internal API right now.

I try to use "react": "16.0.0.-alpha.3" and my error already disappear.
thank for @theohdv and @Chathula

I've the same error!

The issue was in our cases, that ReactComponentWithPureRenderMixin is required for instance by "react-addons-pure-render-mixin" this way:
module.exports = require('react/lib/ReactComponentWithPureRenderMixin');
But the path "node_modules/react/lib" was removed from 0.43.
react-addons-pure-render-mixin gots never being fixed for that.
react-native-router-flux V3 is based on react-native-navigation-experimental that will require itself the react-addons-pure-render-mixin wwhich requires the ReactComponentWithPureRenderMixin that is gone.
Unfortunately the react-native-router-flux goes deprecated recently while the V4-successor is still in beta and not working as a sucessor in all of our apps due to lag of backward compatibility and a lag of documentation (regarding Drawer for instance).
some sources:

https://github.com/aksonov/react-native-router-flux/issues/1974#issuecomment-316026221
https://github.com/aksonov/react-native-router-flux/issues/2045

Thanks @theohdv.
Going with "react": "16.0.0.-alpha.3" instead of  "react": "16.0.0.-alpha.6" did a trick.

Run the following command and it will be solved-
yarn add react-addons-pure-render-mixin
this is manual step to add this module.


Sirius Scalator

Source name
Sirius Scalator
Source link
https://siriusscanlator.blog/
Source language
Português (Brasil)
Other details
18+/NSFW= yes

Acknowledgements

[X] I have checked that the extension does not already exist by searching the extension listing and verified it does not appear there.
[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a meaningful title with the source name.
[X] I will fill out all of the requested information in this form.

Scanlator provides translations and edits on MangaDex.
There is no reading feature on the website.


Fix for name with - in the name

I detected an issue
If the name of the server in the inventory contain minus sign (-) adding it to the inventory failed with no error

Hi @websylv Do you have an example / test case that I can use to build a fix for ?

Hey @websylv  thank you SO much for finding and fixing this issue!
We would need a test for this before I can merge it though. Would you be comfortable writing an additional pester test that does that?
Cheers!


Org boxel to app boxel migration

Proposing adding a migration step, when logging in for the first time / using the app it will :+1:

Check for realms data under the new key
If there is some carry on
If not, check the old key

If in there, copy those to the new key

remove the old account data key (set to an empty object)

Incorporated into #1953


Filter support for nested array (attributes)

I am using the 1.48 version.
I have a list that is created based on a factory function (two different kind of cells)
Let's say the json like this (nested):
[
{ name: "a",age:"12"},
{ name: "b",age:"14"},
{ group: "groupB", people: {[ { name: "z",age:"14"}, { name: "s",age:"16"}] }},
{ name: "c",age:"16"},
{ group: "groupA", people: {[ { name: "e",age:"14"}, { name: "f",age:"16"}] }},
{ name: "d",age:"18"}
]

In the documentation there is no explanation for how to filter a nested result...
// returns entries without nested people
var oFilter = new Filter("name", FilterOperator.EQ, sName),

but how can I get the group back that contains also a Name value ?
// returns nothing
var oFilter = new Filter("people.name", FilterOperator.EQ, sName),

The Filter translates the method calls to oData operator URL's that's why we can't rely on native filtering.
I can see it exists in oDATA V2 by adding a "/". I guess that doesnt exists in the Filter method ?

@kyakubi please check OData V4 Model: Filtering -
https://sapui5.hana.ondemand.com/#/topic/5338bd1f9afb45fb8b2af957c3530e8f

We have V2 oData...

@stephania87 As I understand, what @kyakubi  wants is to use V2 filter like this:
/Suppliers?$filter=Address/City eq 'Redmond'
This URL is described in version 2.0:
http://www.odata.org/documentation/odata-version-2-0/uri-conventions/

@ThePlenkov  thanks :)

what @ThePlenkov said is that possible with the Filter methods or do new ones need to be made ?

So its not possible I guess

Ok, let me try once again :D
Is the described here filtering is what you need based on V2 oDataModel and filters - https://stackoverflow.com/questions/39513124/ui5-odata-service-with-batch-get-with-filters
where you can also use the URL with $filter and a query.

I didn't get it to work with the native filters. So I created a custom filter (need a documentation update on that because its really lacking...
So for anyone that wants to filter through a tree you can use something like this to filter...
let oGroupFilter = new Filter("", oNode => {
   if(oNode.Group){
     var match = oNode.Group.some(People=> {
        if (People.name == sName){ return true;}
      });
      return match;
    }
});


az connectedk8s cli is not working

This is autogenerated. Please review and update as needed.
Describe the bug
Command Name
az connectedk8s
Errors:
mapping values are not allowed here
  in "<unicode string>", line 5, column 15:
      - name: name: Get access to a connected kube ...
                  ^
Traceback (most recent call last):
python3.8/site-packages/knack/cli.py, ln 233, in invoke
    cmd_result = self.invocation.execute(args)
cli/core/commands/__init__.py, ln 589, in execute
    parsed_args = self.parser.parse_args(args)
python3.8/site-packages/knack/parser.py, ln 262, in parse_args
    return super(CLICommandParser, self).parse_args(args)
...
python3.8/site-packages/yaml/scanner.py, ln 577, in fetch_value
    raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
  in "<unicode string>", line 5, column 15:
      - name: name: Get access to a connected kube ...
                  ^

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az connectedk8s {} -n {} -g {} --auth-token

Expected Behavior
Environment Summary
macOS-11.1-x86_64-i386-64bit
Python 3.8.8
Installer: HOMEBREW

azure-cli 2.19.1

Extensions:
k8sconfiguration 0.2.0
connectedk8s 0.3.7
k8s-extension 0.1PP.13

Additional Context

connectedk8s


Figure out a manual way to upload JVM artifacts to some public repo

It's a mitigation of not having #1 done.

For publishing the initial version, everything was done without using Gradle Bintray plugin:

call publishToMavenLocal Gradle target
go to ~/.m2/repository
zip (I tested only on this format, tar.gz doesn't work) the top-level directory found there
create a new version on Bintray
select an option to upload the files
upload the ZIP file there, selecting that it should explode the archive
publish the uploaded files

I don't yet know how to publish some next version. When I was doing it for 1.0.0 - > 0.1.0-beta, I got some messages that it's not possible to overwrite some files. That's why the Gradle Bintray plugin may be necessary to for it, because it has some option like overwrite.

I managed to publish 0.2.0-beta version, but with one difference in the above list of steps. Before zipping, I had to rename all files maven-repo-local.xml to maven-repo.xml. Otherwise, Bintray was saying that the file already exists.

No need - automatic publishing works.


TVPlatform mode : name of device lost at restart

In case of homebridge restart, any modification made to the name of the device in home app is lost (still have to check for entries). The device however remains in the correct room.

seems to work only on local computer, not in docker .

Something like this.filePath = HomebridgeAPI.user.persistPath() should work


utils/json: Replace OkJson with Ruby's core JSON.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew tests with your changes locally?

Removes OkJson from vendor/, as Ruby has included its own JSON parser in corelib since 1.9.x and Homebrew is now guaranteed to be running on 2.x+.
Some unscientific local tests with brew info --json=v1 --all show a pretty substantial performance increase.
With OkJson:
$ time brew info --json=v1 --all >/dev/null

real   0m18.275s
user   0m17.859s
sys    0m0.545s

With core json:
$ time brew info --json=v1 --all >/dev/null

real   0m10.142s
user   0m9.553s
sys    0m0.501s

Nice work here.

Thanks! I also have some further ideas for speeding up JSON generation (I have ulterior motives in mind), but they're more involved.

Thanks! I also have some further ideas for speeding up JSON generation (I have ulterior motives in mind), but they're more involved.

Sounds exciting 😁


Support cyrillic characters in queries

Now it seems like "�������" in sqlectron.

Where is looking like that? Could you please include a screen shot?

Thanks. I will try fix that.

Thank you.


Residual connection after decoder attention?

In the decode_from_codebook_indices method there is a residual connection after the decoder attention:
https://github.com/lucidrains/audiolm-pytorch/blob/d244a6c90b33627bad1e38ae2e20e821e6818253/audiolm_pytorch/soundstream.py#L494
Which is not present in the forward method:
https://github.com/lucidrains/audiolm-pytorch/blob/d244a6c90b33627bad1e38ae2e20e821e6818253/audiolm_pytorch/soundstream.py#L567
Is this correct?
In my case, I found significantly higher audio quality during inference after removing this residual connection (matching training conditions).

@alexdemartos oh gosh yes :man_facepalming: thank you for catching this
https://github.com/lucidrains/audiolm-pytorch/commit/e8ee51f1efb5bb1bba26ec691191763dbece0ecc

@alexdemartos does this mean you have already gotten to sampling from the coarse and fine transformers stages?

thanks for detecting this bug! It helps me a lot

ok, i'm guessing he's slinking back off into the darkness to pen his next paper :laughing:
i'll leave him be


how to disable phpbrew connect pecl

Output
☁  ~  phpbrew ext enable xdebug
PHP Warning:  file_get_contents(http://pecl.php.net/channel.xml): failed to open stream: Operation timed out in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35

Warning: file_get_contents(http://pecl.php.net/channel.xml): failed to open stream: Operation timed out in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
===> Enabling extension xdebug
[*] xdebug extension is enabled.

How to disable phpbrew connecting pecl.php.net? When I enable/disable extensions, phpbrew try to connect pecl.php.net, even the extension is already installed, that's waste too much time; and when I'm offline, it makes error.

I'm on the same boat. I'm using phpbrew to preconfigure the machine that later has no internet connectivity. What is the reasoning behind calling pecl? Is that by design or an oversight? I can contribute but need to know.

hi @mirfilip @0captain  I will take a look at this issue when I have time. thanks for reporting this.

Hi, I'm stuck with this problem too when I try to install pear. I'm behind a proxy, and all in all, it triggers the same warning for the same reason.
In fetchChannelXml(), I see that the method checks the cache before downloading. Is it possible to push the channel.xml in cache "manually" ?

Same problem here. I am behind a proxy, but I haven't had any problem using phpbrew until now. When trying to install xdebug I get the following error:
PHP Warning:  file_get_contents(http://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
Warning: file_get_contents(http://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
PHP Warning:  file_get_contents(https://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
Warning: file_get_contents(https://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
PHP Warning:  file_get_contents(http://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
Warning: file_get_contents(http://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
PHP Warning:  file_get_contents(https://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
Warning: file_get_contents(https://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
PHP Warning:  file_get_contents(http://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
Warning: file_get_contents(http://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
PHP Warning:  file_get_contents(https://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
Warning: file_get_contents(https://pecl.php.net/channel.xml): failed to open stream: Network is unreachable in phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Core.php on line 35
Exception: channel.xml fetch failed.
Thrown from phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Channel.php at line 148:
145        }
146
147        if( ! $xmlstr ) {

148            throw new Exception('channel.xml fetch failed.');
149        }
150
151        // save cache
152        if( $this->cache ) {

Trace:
0) PEARX\Channel->fetchChannelXml('pecl.php.net')
    from phar:///usr/local/bin/phpbrew/vendor/corneltek/pearx/src/PEARX/Channel.php: 64

1) PEARX\Channel->__construct('pecl.php.net')
    from phar:///usr/local/bin/phpbrew/src/PhpBrew/Extension/Provider/PeclProvider.php: 37

2) PhpBrew\Extension\Provider\PeclProvider->getPackageXml('xdebug', 'stable')
    from phar:///usr/local/bin/phpbrew/src/PhpBrew/Extension/Provider/PeclProvider.php: 74

3) PhpBrew\Extension\Provider\PeclProvider->buildPackageDownloadUrl('stable')
    from phar:///usr/local/bin/phpbrew/src/PhpBrew/Extension/ExtensionDownloader.php: 35

4) PhpBrew\Extension\ExtensionDownloader->download(PhpBrew\Extension\Provider\PeclProvider, 'stable')
    from phar:///usr/local/bin/phpbrew/src/PhpBrew/Command/ExtensionCommand/InstallCommand.php: 169

5) PhpBrew\Command\ExtensionCommand\InstallCommand->execute('xdebug')

PHP Notice:  Undefined index: file in phar:///usr/local/bin/phpbrew/vendor/corneltek/cliframework/src/ExceptionPrinter/DevelopmentExceptionPrinter.php on line 101
Notice: Undefined index: file in phar:///usr/local/bin/phpbrew/vendor/corneltek/cliframework/src/ExceptionPrinter/DevelopmentExceptionPrinter.php on line 101
PHP Notice:  Undefined index: line in phar:///usr/local/bin/phpbrew/vendor/corneltek/cliframework/src/ExceptionPrinter/DevelopmentExceptionPrinter.php on line 101
Notice: Undefined index: line in phar:///usr/local/bin/phpbrew/vendor/corneltek/cliframework/src/ExceptionPrinter/DevelopmentExceptionPrinter.php on line 101
from : 0
6) call_user_func_array([PhpBrew\Command\ExtensionCommand\InstallCommand, 'execute'], ['xdebug'])
    from phar:///usr/local/bin/phpbrew/vendor/corneltek/cliframework/src/CommandBase.php: 845

7) CLIFramework\CommandBase->executeWrapper(['xdebug'])
    from phar:///usr/local/bin/phpbrew/vendor/corneltek/cliframework/src/Application.php: 398

8) CLIFramework\Application->run(['/usr/local/bin/phpbrew', '-d', 'ext', 'install', '--http-proxy=myproxy.com:8080', 'xdebug'])
    from phar:///usr/local/bin/phpbrew/src/PhpBrew/Console.php: 119

9) PhpBrew\Console->runWithTry(['/usr/local/bin/phpbrew', '-d', 'ext', 'install', '--http-proxy=myproxy.com:8080', 'xdebug'])
    from phar:///usr/local/bin/phpbrew/scripts/phpbrew-emb.php: 2

10) require('phar:///usr/local/bin/phpbrew/scripts/phpbrew-emb.php')
    from /usr/local/bin/phpbrew: 99

Is there any known issue about this? Thank you!


JWT token and hashed password in MySQL Workbench

[x] JWT token;
[x] Hashed password in MySQL Workbench;

The JWT now has expiration time.


[SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

What changes were proposed in this pull request?
Spark will request yarn cluster metrics and print a log about nodemanager number, it's not so important and this rpc is always slow

We can make it as debug level
Why are the changes needed?
Avoid unnecessary delay when submit application.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Not need

ping @tgravescs @mridulm

Test build #146473 has started for PR 34982 at commit bafb1ce.

Test build #146473 has finished for PR 34982 at commit bafb1ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146473/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50949/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50949/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50949/

Also ping @srowen

ping @tgravescs

I guess I'm fine with this, I kind of assume this is because your yarn resource manager is overloaded or unresponsive though again?

Yea, yarn's performance is not so stable. With this patch, performance become better ==

Gentle ping

ping @srowen @tgravescs

Merging to master.


Option to Omit File and Parameter data from Export.

First off, thanks so much for the Extract Data Extension for Moneydance--very useful!
I am using exported .csv file as an ODBC data source for reporting in Excel.  For whatever reason, the driver I am using does not properly recognize the last data row in the .csv file and does some funky stuff on the rows containing the file and parameter Information.  Might it be possible to have an option to NOT export these additional rows and ONLY include the data rows--something like this?

Good idea and with a good reason... OK, I have now done this.. Please give it a try. Can you do me a favour and test all the extracts to ensure they all work (without crashing) on your setup.. I also found and fixed a tiny (new) bug, and also improved the Account Drop down list display. You can download latest from my site. Let me know? Thanks

OK, I checked basic functionality with no parameters selected as follows:

[x] StockGlance2020 with parameters -- no crash, file has parameters, display does not
[x] StockGlance2020 without parameters -- no crash, file and display do not have parameters
[x] Reminders with parameters -- no crash, file has parameters, display does not
[x] Reminders without parameters -- no crash, file and display do not have parameters
[x] Account register transactions with parameters -- no crash, file has parameters
[x] Account register transactions without parameters -- no crash, file does not have parameters.
[x] Investment transactions with parameters -- no crash, file has parameters.
[x] Investment transactions without parameters -- no crash, file does not have parameters.
[x] Currency price history with parameters -- no crash, file has parameters
[x] Currency price history without parameters -- no crash, file does not have parameters

So it all checks out. Great job, thanks!
I did run into one other issue with the way I am using the data from the extracts.  I will post this as a new issue.

Thanks!


Add thread sanitizer support

@swift-ci please smoke test Linux

@swift-ci please smoke test

@swift-ci please smoke test

@swift-ci please smoke test

@aciidb0mb3r It looks like we have a failure in swift-4.2-branch testing as a result of this.
https://ci.swift.org/view/swift-4.2-branch/job/oss-swift-4.2-incremental-RA-osx/255/

But trunk seems to work. Maybe something is broken in sanitizers on 4.2?

@aciidb0mb3r I'm not sure. I believe this will block PR testing for that branch though.

Yeah. I'll disable the test for now and file a bug for investigation.

https://github.com/apple/swift-package-manager/commit/0fbfa19c89856dc155234f50a6ffd1f86f4af3ae

Thanks!


fixed the Unsubscribe Button: Not Margined (Issue #26835)

added padding-right to subscriber_list_remove class
Fixes: #26835

Screenshots and screen captures:

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

@alya could you please review this.

Thanks for the fix! Could you please update your commit message to match the commit style guidelines?

@alya I have updated the commit message, is it alright now ?

@alya, I have once again edited the commit message, is it now in accordance to the guidelines ?

Yes, that looks right.
@sahil839 Could you please review this one after 8.0?

@zulipbot claim

Looks good to me.
You would need to update the commit message to have "Fixes" line as a separate line. Please read https://zulip.readthedocs.io/en/latest/contributing/commit-discipline.html#commit-messages carefully and update the commit message accordingly.

@sahil839
Some backend checks were failing, I am pretty sure it is not because of my commit, do you want me to look into it or is it fine for now

Yeah, the CI failure is not related to your changes.
But the commit message still needs changes -

both lines should finish with a period.
And the second line should be "Fixes #26835.".

Please make sure to read the guidelines carefully next time.

Yeah, the CI failure is not related to your changes.
But the commit message still needs changes -

both lines should finish with a period.
And the second line should be Fixes #26835..

Please make sure to read the guidelines carefully next time.

Yeah, the CI failure is not related to your changes.
But the commit message still needs changes -

both lines should finish with a period.
And the second line should be Fixes #26835..

Please make sure to read the guidelines carefully next time.

Will do @sahil839 , I have re written the commit, only thing is that after typing # in the description, it is automatically appending  zulip before it, I have read the documentation but there is no example given whether it should look like this only.

That's fine since it is added by Github.
@timabbott this is ready for your review. The test failure is unrelated and is being discussed in https://chat.zulip.org/#narrow/stream/43-automated-testing/topic/flaky.20test_remote_billing.20test.

Merged, thanks @Ayush2k02 and @sahil839! Sorry for the slow review, this got lost over the holidays in part due to failing CI; in the future, please be encouraged to rebase your work if it's failing tests due to something that's not your fault.


implemented tin

Nice, looks good! How would you feel about trying to create a test for this one?

TIN!

Relates to #83


how use skip?

You use skip to determine that should some of specific request does not need to be counted into rate-limiting. The documentation is already self-explanatory.


Added Tower Of Hanoi game

PR Description 📜
I have added the HTML, CSS and JavaScript files for the game. I have also created a README according to the template given in the repository.
Fixes #1411

Mark the task you have completed ✅

[x] I follow CONTRIBUTING GUIDELINE & CODE OF CONDUCT of this project.
[x] I have performed a self-review of my own code or work.
[x] My changes generates no new warnings.
[x] I have followed proper naming convention showed in CONTRIBUTING GUIDELINE
[x] I have added screenshot for website preview in assets/images
[x] I have added entries for my game in main README.md
[x] I have added README.md in my folder
[x] I have specified the respective issue number for which I have requested the new game.
[x] I have added working video of the game in README.md (optional)

Thank you soo much for contributing to our repository 💗

Thank you @AnkitaBarbora ,for creating the PR and contributing to our GameZone 💗
Review team will review the PR and will reach out to you soon! 😇
Make sure that you have marked all the tasks that you are done with ✅.
Thank you for your patience! 😀

Thank you  @AnkitaBarbora , for your valuable time and contribution in our GameZone 💗.
It’s our GameZone, so Let’s build this GameZone altogether !!🤝
Hoping to see you soon with another PR again 😇
Wishing you all the best for your journey into Open Source🚀

Hey @AnkitaBarbora,
You can't work on a closed issue 👀!
Follow the proper guideline and make a new PR again 😀.
Happy Hacking 💗

Okay Sir, thank you.


App constraints handling in create/edit dialog broken

Affects: latest build
Steps to reproduce:

open the create dialog and fill in a simple app definition
add an invalid constraint, e.g. foo

Result: no request is posted, but no ui-error is shown either

change the constraint to hostname:unique (considered invalid)

Result: Error: App creation unsuccessful. Check your app settings and try again.

change the constraint to hostname:UNIQUE (which actually is valid)

Result: the app is created (I sometimes had to press create twice, though)

now, open the app configuration and inspect the constraints section.

Result: the constraint is rendered as hostname,UNIQUE. This cannot be saved and does not show errors when hitting create

Entering a space in the URIs field causes the value uris=[""] to be sent, which delays the app forever.


Mapping to scvs

https://scvs.owasp.org/scvs/v5-component-analysis/

Hi @prabhu ,
I can we can map for 5.6. 5.9, 5.10, 5.11, 5.12 because they are directly related to SBOM generation path

Some requirements can also be map from https://scvs.owasp.org/scvs/v2-software-bill-of-materials/

We can also target https://scvs.owasp.org/bom-maturity-model/ to create a maturity report once the SBOM is generated


Load watch_history_dashboard.json failed

ELK version: 5.4.1
I followed this guide:
https://github.com/elastic/examples/tree/master/Alerting/watcher_dashboard
When I import the json file, it always report a error:
Saved Objects: The file could not be processed.

This issue was cross-posted on Discus: https://discuss.elastic.co/t/load-json-file-failed-the-file-could-not-be-processed/94107/2
I verified the ability to import the saved objects and am closing this issue.


Many failed logical tests

About You:
Name: Max
Company: GigaSpaces
Your question:
Hello. We are testing our connector with TDVT and see that many tests are failing.
Below is an example of a failing query.
I am trying to understand the reason for this test failure, can you kindly advice what can be the reason for this failure or help me to shorten the query in order to nail down the problem.
Please see the original query and the error at the bottom.
SELECT CAST(CONCAT('1234.', CASE WHEN 2 >= 0 THEN RIGHT("Calcs"."key",2) ELSE NULL END) AS FLOAT) AS "TEMP(Test)(93928613)(0)" FROM "TestV1"."Calcs" "Calcs" GROUP BY 1
For validation, this one works:
SELECT "Calcs"."key" FROM "TestV1"."Calcs" "Calcs"
key
key01
key07
key16
key10
key12
key03
key14
key09
key05
key04
key02
key00
key11
key06
key08
key15
key13
2020-03-26 10:28:26,302  SEVERE [com.tableausoftware.jdbc.JDBCProtocolImpl] - Exception in runQuery for query: SELECT CAST(CONCAT('1234.', CASE WHEN 2 >= 0 THEN RIGHT("Calcs"."key",2) ELSE NULL END) AS FLOAT) AS "TEMP(Test)(93928613)(0)"
FROM "TestV1"."Calcs" "Calcs"
GROUP BY 1. Error while executing SQL "SELECT CAST(CONCAT('1234.', CASE WHEN 2 >= 0 THEN RIGHT("Calcs"."key",2) ELSE NULL END) AS FLOAT) AS "TEMP(Test)(93928613)(0)"
FROM "TestV1"."Calcs" "Calcs"
GROUP BY 1": From line 1, column 51 to line 1, column 72: No match found for function signature RIGHT(, )
Thanks
Max

Hi Max, For this query, your error message is  No match found for function signature RIGHT(, ) You will need to create RIGHT() function with currently available functions for your database in the dialect file.

hey @MaxB-Giga any update on testing for this ? Thanks !

tfs 1197816

Hello @jainam1995
This issue can be closed, I didn't hear about reoccurrence.
Thanks and regards
Max

Closing


Add to_bcolz method to dask.dataframe

Since we have a from_bcolz function, we should add a corresponding to_bcolz method.
From https://github.com/dask/dask/pull/1366.

Should this support ctable (that's what from_bcolz handles currently) only or carray as well?

Perhaps Series would map to a carray?

I tested in my forked dask environment to_bcolz  (based off to_hdf + https://github.com/visualfabriq/bquery/blob/master/bquery/benchmarks/taxi/load.py) and it seems to work
in dask/dataframe/core.py
def to_bcolz(self, outfile):
from .io import to_bcolz
return to_bcolz(self, outfile)
in dask/dataframe/io.py
import bcolz
def _pd_to_bcolz(pd_to_bcolz, lock, args, kwargs=None):
if lock:
lock.acquire()
try:
if kwargs['mode']=='w':
pd_to_bcolz(args[0], rootdir=args[1], mode='w') # add expectedlen=outfile_Nrows
else:
import_ct = bcolz.ctable(rootdir=args[1], mode='a')
temp_ct = bcolz.ctable.fromdataframe(args[0])
import_ct.append(temp_ct)
finally:
if lock:
lock.release()
return None

def to_bcolz(df, outfile):
name = 'to-bcolz-' + uuid.uuid1().hex
pd_to_bcolz = bcolz.ctable.fromdataframe
kwargs={'mode':'w'}
dask_kwargs={}
lock=False
compute=True
name_function=None

fmt_obj = lambda outfile, _: outfile
if name_function is None: name_function = build_name_function(df.npartitions - 1)
i_name = name_function(0)

dsk = dict()
dsk[(name, 0)] = (_pd_to_bcolz, pd_to_bcolz,
                  lock,
                  [(df._name, 0), fmt_obj(outfile, i_name)],
                  kwargs)

kwargs2 = kwargs.copy()
kwargs2['mode'] = 'a'
for i in range(1, df.npartitions):
    i_name = name_function(i)
    task = (_pd_to_bcolz, pd_to_bcolz, lock,
            [(df._name, i), fmt_obj(outfile, i_name)], kwargs2)

    link_dep = i - 1
    task = (_link, (name, link_dep), task)
    dsk[(name, i)] = task

dsk = merge(df.dask, dsk)
keys = [(name, df.npartitions - 1)]

if compute:
    return DataFrame._get(dsk, keys, get=dask.get, **dask_kwargs)
else:
    return delayed([Delayed(key, [dsk]) for key in keys])

I tested in my forked dask environment to_bcolz (based off to_hdf +
https://github.com/visualfabriq/bquery/blob/master/bquery/benchmarks/taxi/load.py)
and it seems to work
##in dask/dataframe/core.py
def to_bcolz(self, outfile):
from .io import to_bcolz
return to_bcolz(self, outfile)
##in dask/dataframe/io.py
import bcolz
def _pd_to_bcolz(pd_to_bcolz, lock, args, kwargs=None):
if lock:
lock.acquire()
try:
if kwargs['mode']=='w':
pd_to_bcolz(args[0], rootdir=args[1], mode='w') # add
expectedlen=outfile_Nrows
else:
import_ct = bcolz.ctable(rootdir=args[1], mode='a')
temp_ct = bcolz.ctable.fromdataframe(args[0])
import_ct.append(temp_ct)
finally:
if lock:
lock.release()
return None

def to_bcolz(df, outfile):
name = 'to-bcolz-' + uuid.uuid1().hex
pd_to_bcolz = bcolz.ctable.fromdataframe
kwargs={'mode':'w'}
dask_kwargs={}
lock=False
compute=True
name_function=None

fmt_obj = lambda outfile, _: outfile
if name_function is None: name_function =

build_name_function(df.npartitions - 1)
i_name = name_function(0)
dsk = dict()
dsk[(name, 0)] = (_pd_to_bcolz, pd_to_bcolz,
                  lock,
                  [(df._name, 0), fmt_obj(outfile, i_name)],
                  kwargs)

kwargs2 = kwargs.copy()
kwargs2['mode'] = 'a'
for i in range(1, df.npartitions):
    i_name = name_function(i)
    task = (_pd_to_bcolz, pd_to_bcolz, lock,
            [(df._name, i), fmt_obj(outfile, i_name)], kwargs2)

    link_dep = i - 1
    task = (_link, (name, link_dep), task)
    dsk[(name, i)] = task

dsk = merge(df.dask, dsk)
keys = [(name, df.npartitions - 1)]

if compute:
    return DataFrame._get(dsk, keys, get=dask.get, **dask_kwargs)
else:
    return delayed([Delayed(key, [dsk]) for key in keys])

On Wed, Jul 13, 2016 at 5:20 AM, Matthew Rocklin<EMAIL_ADDRESS>wrote:

Perhaps Series would map to a carray?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1368#issuecomment-232338275, or mute
the thread
https://github.com/notifications/unsubscribe/AJmNzYjpcLxr30R0CmPxcKq7AFN7_NYMks5qVNf9gaJpZM4JJfAx
.

ok, here is a test
(dask_dev) [akravchenko@hnode1 ~]$ ipython
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
Type "copyright", "credits" or "license" for more information.
In [1]: import dask.dataframe as dd
In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: import random, bcolz
In [5]:
In [5]: t0=pd.DataFrame({'v0':np.random.randint(0,1000,(1000000,))})
In [6]: t0_d = dd.from_pandas(t0, npartitions=10)
In [7]: t0_d
Out[7]: dd.DataFrame<from_pa..., npartitions=10, divisions=(0, 100000, 200000, ..., 900000, 999999)>
In [8]: t0_d.to_bcolz(r'/mnt/outfile.bcolz')
Out[8]: (None,)
In [9]: t1 = dd.from_bcolz(r'/mnt/outfile.bcolz', blocksize=1000000, lock=False)
In [10]: t1
Out[10]: dd.DataFrame<from_bc..., npartitions=31, divisions=(0, 32768, 65536, ..., 983040, 999999)>
In [11]:

ok, here is a test
(dask_dev) [akravchenko@hnode1 ~]$ ipython
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
Type "copyright", "credits" or "license" for more information.
In [1]: import dask.dataframe as dd
In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: import random, bcolz
In [5]:
In [5]: t0=pd.DataFrame({'v0':np.random.randint(0,1000,(1000000,))})
In [6]: t0_d = dd.from_pandas(t0, npartitions=10)
In [7]: t0_d
Out[7]: dd.DataFrame<from_pa..., npartitions=10, divisions=(0, 100000,
200000, ..., 900000, 999999)>
In [8]: t0_d.to_bcolz(r'/mnt/outfile.bcolz')
Out[8]: (None,)
In [9]: t1 = dd.from_bcolz(r'/mnt/outfile.bcolz', blocksize=1000000,
lock=False)
In [10]: t1
Out[10]: dd.DataFrame<from_bc..., npartitions=31, divisions=(0, 32768,
65536, ..., 983040, 999999)>
On Wed, Jul 13, 2016 at 9:53 AM, Nir<EMAIL_ADDRESS>wrote:

@ankravch https://github.com/ankravch Could you perhaps create a PR for
a branch with those changes?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1368#issuecomment-232417185, or mute
the thread
https://github.com/notifications/unsubscribe/AJmNzYf2POPnjbm-FD--G09ub1kuDyTGks5qVRf2gaJpZM4JJfAx
.

ok, I think my current version of to_bcolz does not work correctly, here is
my test on a single machine:
import dask.dataframe as dd
import bcolz, dask
from dask.distributed import Executor, progress
from dask.diagnostics import ProgressBar
dgn_table = dd.from_bcolz(r'/mnt/nfs/test_dgn_pb_200M.bcolz',
blocksize=1000000, lock=False)
dgn_finder = dd.from_bcolz(r'/mnt/nfs/test_dgn9_finder_2.bcolz',
blocksize=1000000, lock=False)
[########################################] | 100% Completed | 31.2s
[########################################] | 100% Completed |  0.1s
#* ~200% (out of 400) CPU is used*
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_hdf(r'/mnt/nfs/test_tohdf_link_5.h5', 'test',
format='table', mode='w'))
[########################################] | 100% Completed |  2min 21.5s
# only ~100% (out of 400) CPU is used
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_bcolz(r'/mnt/nfs/test_tobcolz_link_5.bcolz'))
[########################################] | 100% Completed |  2min 14.1s
On Wed, Jul 13, 2016 at 1:49 PM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

ok, here is a test
(dask_dev) [akravchenko@hnode1 ~]$ ipython
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
Type "copyright", "credits" or "license" for more information.
In [1]: import dask.dataframe as dd
In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: import random, bcolz
In [5]:
In [5]: t0=pd.DataFrame({'v0':np.random.randint(0,1000,(1000000,))})
In [6]: t0_d = dd.from_pandas(t0, npartitions=10)
In [7]: t0_d
Out[7]: dd.DataFrame<from_pa..., npartitions=10, divisions=(0, 100000,
200000, ..., 900000, 999999)>
In [8]: t0_d.to_bcolz(r'/mnt/outfile.bcolz')
Out[8]: (None,)
In [9]: t1 = dd.from_bcolz(r'/mnt/outfile.bcolz', blocksize=1000000,
lock=False)
In [10]: t1
Out[10]: dd.DataFrame<from_bc..., npartitions=31, divisions=(0, 32768,
65536, ..., 983040, 999999)>
On Wed, Jul 13, 2016 at 9:53 AM, Nir<EMAIL_ADDRESS>wrote:

@ankravch https://github.com/ankravch Could you perhaps create a PR
for a branch with those changes?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1368#issuecomment-232417185, or mute
the thread
https://github.com/notifications/unsubscribe/AJmNzYf2POPnjbm-FD--G09ub1kuDyTGks5qVRf2gaJpZM4JJfAx
.

when I run with get=dask.async.get_sync:
join+to_hdf, cpu ~200% (out of 400)
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_hdf(r'/mnt/nfs/test_tohdf_link_5.h5', 'test',
format='table', mode='w'), get=dask.async.get_sync)
[########################################] | 100% Completed |  2min 22.1s
only join, cpu usage ~100%  (out of 400)
res=dask.compute(t0.count(),get=dask.async.get_sync)
[########################################] | 100% Completed |  1min 53.2s
(CLAIMNO    71303395
dtype: int64,)
join+to_bcolz, cpu usage ~100% (out of 400)
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_bcolz(r'/mnt/nfs/test_tobcolz_link_5.bcolz'),
get=dask.async.get_sync)
[########################################] | 100% Completed |  2min 13.4s
I guess my question is how join+to_hdf (in dask.async.get_sync mode)
makes 200% CPU ?
On Wed, Jul 13, 2016 at 4:56 PM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

ok, I think my current version of to_bcolz does not work correctly, here
is my test on a single machine:
import dask.dataframe as dd
import bcolz, dask
from dask.distributed import Executor, progress
from dask.diagnostics import ProgressBar
dgn_table = dd.from_bcolz(r'/mnt/nfs/test_dgn_pb_200M.bcolz',
blocksize=1000000, lock=False)
dgn_finder = dd.from_bcolz(r'/mnt/nfs/test_dgn9_finder_2.bcolz',
blocksize=1000000, lock=False)
[########################################] | 100% Completed | 31.2s
[########################################] | 100% Completed |  0.1s
#* ~200% (out of 400) CPU is used*
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_hdf(r'/mnt/nfs/test_tohdf_link_5.h5', 'test',
format='table', mode='w'))
[########################################] | 100% Completed |  2min 21.5s
# only ~100% (out of 400) CPU is used
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN', how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_bcolz(r'/mnt/nfs/test_tobcolz_link_5.bcolz'))
[########################################] | 100% Completed |  2min 14.1s
On Wed, Jul 13, 2016 at 1:49 PM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

ok, here is a test
(dask_dev) [akravchenko@hnode1 ~]$ ipython
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
Type "copyright", "credits" or "license" for more information.
In [1]: import dask.dataframe as dd
In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: import random, bcolz
In [5]:
In [5]: t0=pd.DataFrame({'v0':np.random.randint(0,1000,(1000000,))})
In [6]: t0_d = dd.from_pandas(t0, npartitions=10)
In [7]: t0_d
Out[7]: dd.DataFrame<from_pa..., npartitions=10, divisions=(0, 100000,
200000, ..., 900000, 999999)>
In [8]: t0_d.to_bcolz(r'/mnt/outfile.bcolz')
Out[8]: (None,)
In [9]: t1 = dd.from_bcolz(r'/mnt/outfile.bcolz', blocksize=1000000,
lock=False)
In [10]: t1
Out[10]: dd.DataFrame<from_bc..., npartitions=31, divisions=(0, 32768,
65536, ..., 983040, 999999)>
On Wed, Jul 13, 2016 at 9:53 AM, Nir<EMAIL_ADDRESS>wrote:

@ankravch https://github.com/ankravch Could you perhaps create a PR
for a branch with those changes?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1368#issuecomment-232417185, or mute
the thread
https://github.com/notifications/unsubscribe/AJmNzYf2POPnjbm-FD--G09ub1kuDyTGks5qVRf2gaJpZM4JJfAx
.

using dask.dataframe and dumping output dataframe in ram
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0, get=dask.async.get_sync)
print (res[0])
[########################################] | 100% Completed |  1min 53.2s
(71303395, 1)
and than using pandas to save dask output dataframe (res[0]) as h5/bcolz:
res[0].to_hdf(r'/mnt/nfs/test_res.h5', 'test', format='table', mode='w')
0:01:10.806566
res[0].to_hdf(r'/mnt/nfs/test_res.h5', 'test', mode='w') # cons -
cannot load back to dask.dateframe
0:00:03.216095
bcolz.ctable.fromdataframe(res[0], rootdir=r'/mnt/nfs/test_res.bcolz')
0:00:01.696645
Ideally I would want to see dask to_bcolz function
that preserves ~same performance (as bcolz.ctable.fromdataframe using
pandas) and and uses available ram wisely. Do you think it's doable?
On Thu, Jul 14, 2016 at 8:06 AM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

when I run with get=dask.async.get_sync:
join+to_hdf, cpu ~200% (out of 400)
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_hdf(r'/mnt/nfs/test_tohdf_link_5.h5', 'test',
format='table', mode='w'), get=dask.async.get_sync)
[########################################] | 100% Completed |  2min 22.1s
only join, cpu usage ~100%  (out of 400)
res=dask.compute(t0.count(),get=dask.async.get_sync)
[########################################] | 100% Completed |  1min 53.2s
(CLAIMNO    71303395
dtype: int64,)
join+to_bcolz, cpu usage ~100% (out of 400)
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN', how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_bcolz(r'/mnt/nfs/test_tobcolz_link_5.bcolz'), get=dask.async.get_sync)
[########################################] | 100% Completed |  2min 13.4s
I guess my question is how join+to_hdf (in dask.async.get_sync mode) makes 200% CPU ?
On Wed, Jul 13, 2016 at 4:56 PM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

ok, I think my current version of to_bcolz does not work correctly, here
is my test on a single machine:
import dask.dataframe as dd
import bcolz, dask
from dask.distributed import Executor, progress
from dask.diagnostics import ProgressBar
dgn_table = dd.from_bcolz(r'/mnt/nfs/test_dgn_pb_200M.bcolz',
blocksize=1000000, lock=False)
dgn_finder = dd.from_bcolz(r'/mnt/nfs/test_dgn9_finder_2.bcolz',
blocksize=1000000, lock=False)
[########################################] | 100% Completed | 31.2s
[########################################] | 100% Completed |  0.1s
#* ~200% (out of 400) CPU is used*
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_hdf(r'/mnt/nfs/test_tohdf_link_5.h5', 'test',
format='table', mode='w'))
[########################################] | 100% Completed |  2min 21.5s
# only ~100% (out of 400) CPU is used
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN', how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_bcolz(r'/mnt/nfs/test_tobcolz_link_5.bcolz'))
[########################################] | 100% Completed |  2min 14.1s
On Wed, Jul 13, 2016 at 1:49 PM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

ok, here is a test
(dask_dev) [akravchenko@hnode1 ~]$ ipython
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
Type "copyright", "credits" or "license" for more information.
In [1]: import dask.dataframe as dd
In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: import random, bcolz
In [5]:
In [5]: t0=pd.DataFrame({'v0':np.random.randint(0,1000,(1000000,))})
In [6]: t0_d = dd.from_pandas(t0, npartitions=10)
In [7]: t0_d
Out[7]: dd.DataFrame<from_pa..., npartitions=10, divisions=(0, 100000,
200000, ..., 900000, 999999)>
In [8]: t0_d.to_bcolz(r'/mnt/outfile.bcolz')
Out[8]: (None,)
In [9]: t1 = dd.from_bcolz(r'/mnt/outfile.bcolz', blocksize=1000000,
lock=False)
In [10]: t1
Out[10]: dd.DataFrame<from_bc..., npartitions=31, divisions=(0, 32768,
65536, ..., 983040, 999999)>
On Wed, Jul 13, 2016 at 9:53 AM, Nir<EMAIL_ADDRESS>wrote:

@ankravch https://github.com/ankravch Could you perhaps create a PR
for a branch with those changes?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1368#issuecomment-232417185, or mute
the thread
https://github.com/notifications/unsubscribe/AJmNzYf2POPnjbm-FD--G09ub1kuDyTGks5qVRf2gaJpZM4JJfAx
.

ok, here is a follow up
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0, get=dask.async.get_sync)
print (res[0].shape)
back to dask.dateframe
dres = dd.from_pandas(res[0], npartitions=1)
dres.to_hdf(r'/mnt/nfs/test_res.h5', 'test', format='table', mode='w')
[########################################] | 100% Completed | 31.5s
dres.to_bcolz(r'/mnt/nfs/test_res.bcolz')
[########################################] | 100% Completed |  2.9s
dask to_bcolz performs acceptable, but it brings me back to the
original question:
dask join+to_hdf, cpu ~200% (out of 400), runtime=2min 22s
dask join+to_bcolz, cpu usage ~100% (out of 400), run time=2min 13s
how join+to_hdf (in dask.async.get_sync mode) makes 200% CPU
On Thu, Jul 14, 2016 at 8:46 AM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

using dask.dataframe and dumping output dataframe in ram
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0, get=dask.async.get_sync)
print (res[0])
[########################################] | 100% Completed |  1min 53.2s
(71303395, 1)
and than using pandas to save dask output dataframe (res[0]) as h5/bcolz:
res[0].to_hdf(r'/mnt/nfs/test_res.h5', 'test', format='table', mode='w')
0:01:10.806566
res[0].to_hdf(r'/mnt/nfs/test_res.h5', 'test', mode='w') # cons - cannot load back to dask.dateframe
0:00:03.216095
bcolz.ctable.fromdataframe(res[0], rootdir=r'/mnt/nfs/test_res.bcolz')
0:00:01.696645
Ideally I would want to see dask to_bcolz function
that preserves ~same performance (as bcolz.ctable.fromdataframe using pandas) and and uses available ram wisely. Do you think it's doable?
On Thu, Jul 14, 2016 at 8:06 AM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

when I run with get=dask.async.get_sync:
join+to_hdf, cpu ~200% (out of 400)
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_hdf(r'/mnt/nfs/test_tohdf_link_5.h5', 'test',
format='table', mode='w'), get=dask.async.get_sync)
[########################################] | 100% Completed |  2min 22.1s
only join, cpu usage ~100%  (out of 400)
res=dask.compute(t0.count(),get=dask.async.get_sync)
[########################################] | 100% Completed |  1min 53.2s
(CLAIMNO    71303395
dtype: int64,)
join+to_bcolz, cpu usage ~100% (out of 400)
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN', how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_bcolz(r'/mnt/nfs/test_tobcolz_link_5.bcolz'), get=dask.async.get_sync)
[########################################] | 100% Completed |  2min 13.4s
I guess my question is how join+to_hdf (in dask.async.get_sync mode) makes 200% CPU ?
On Wed, Jul 13, 2016 at 4:56 PM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

ok, I think my current version of to_bcolz does not work correctly, here
is my test on a single machine:
import dask.dataframe as dd
import bcolz, dask
from dask.distributed import Executor, progress
from dask.diagnostics import ProgressBar
dgn_table = dd.from_bcolz(r'/mnt/nfs/test_dgn_pb_200M.bcolz',
blocksize=1000000, lock=False)
dgn_finder = dd.from_bcolz(r'/mnt/nfs/test_dgn9_finder_2.bcolz',
blocksize=1000000, lock=False)
[########################################] | 100% Completed | 31.2s
[########################################] | 100% Completed |  0.1s
#* ~200% (out of 400) CPU is used*
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN',
how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_hdf(r'/mnt/nfs/test_tohdf_link_5.h5', 'test',
format='table', mode='w'))
[########################################] | 100% Completed |  2min 21.5s
# only ~100% (out of 400) CPU is used
t0=dd.merge(dgn_table[['DGN','CLAIMNO']], dgn_finder, on='DGN', how='inner')[['CLAIMNO']]
res=dask.compute(t0.to_bcolz(r'/mnt/nfs/test_tobcolz_link_5.bcolz'))
[########################################] | 100% Completed |  2min 14.1s
On Wed, Jul 13, 2016 at 1:49 PM, Anton Kravchenko <
<EMAIL_ADDRESS>wrote:

ok, here is a test
(dask_dev) [akravchenko@hnode1 ~]$ ipython
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
Type "copyright", "credits" or "license" for more information.
In [1]: import dask.dataframe as dd
In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: import random, bcolz
In [5]:
In [5]: t0=pd.DataFrame({'v0':np.random.randint(0,1000,(1000000,))})
In [6]: t0_d = dd.from_pandas(t0, npartitions=10)
In [7]: t0_d
Out[7]: dd.DataFrame<from_pa..., npartitions=10, divisions=(0, 100000,
200000, ..., 900000, 999999)>
In [8]: t0_d.to_bcolz(r'/mnt/outfile.bcolz')
Out[8]: (None,)
In [9]: t1 = dd.from_bcolz(r'/mnt/outfile.bcolz', blocksize=1000000,
lock=False)
In [10]: t1
Out[10]: dd.DataFrame<from_bc..., npartitions=31, divisions=(0, 32768,
65536, ..., 983040, 999999)>
On Wed, Jul 13, 2016 at 9:53 AM, Nir<EMAIL_ADDRESS>wrote:

@ankravch https://github.com/ankravch Could you perhaps create a PR
for a branch with those changes?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1368#issuecomment-232417185, or mute
the thread
https://github.com/notifications/unsubscribe/AJmNzYf2POPnjbm-FD--G09ub1kuDyTGks5qVRf2gaJpZM4JJfAx
.

any updates on the to_bcolz functionality?

To the best of my knowledge this issue is up to date.  Is this something that you would like to take on @sfragosorodrig ?

Are people still using bcolz?  Most people I interact with today use Parquet, which since the opening of this issue has grown a couple of good reader/writers in Python.  I'm going to close this now.  If this is still an active issue then please ping and I'll reopen.  However I suspect that bcolz my become out-of-scope for dask.dataframe in the future due to low usage.


Utilize path name slug to handle multi-tenancy over using subdomains to avoid having to manage the subdomain routes.

I suspect this will make things easier in the beginning and we can always move forward to using subdomains when the need arises.

Closed with #179


iOS category Conflicts with react-native ART.

Whenever I build my react-native project I get the following Warnings:
ld: warning: meta method 'CGGradient:offset:' in category from /Users/SudoPlz/Library/Developer/Xcode/DerivedData/App-fawgawfawfawfa/Build/Products/Debug-iphonesimulator/libRNSVG.a(RCTConvert+RNSVG.o) conflicts with same method from another category

ld: warning: meta method 'CGColor:offset:' in category from /Users/SudoPlz/Library/Developer/Xcode/DerivedData/App-fawgawfawfawfa/Build/Products/Debug-iphonesimulator/libRNSVG.a(RCTConvert+RNSVG.o) conflicts with same method from another category

ld: warning: meta method 'CGRect:offset:' in category from /Users/SudoPlz/Library/Developer/Xcode/DerivedData/App-fawgawfawfawfa/Build/Products/Debug-iphonesimulator/libRNSVG.a(RCTConvert+RNSVG.o) conflicts with same method from another category

ld: warning: meta method 'CTTextAlignment:' in category from /Users/idynopia/Library/Developer/Xcode/DerivedData/AcuityApp-fabslqdjhkkvxcdcmkayrouftese/Build/Products/Debug-iphonesimulator/libRNSVG.a(RCTConvert+RNSVG.o) conflicts with same method from another category

ld: warning: meta method 'CGPath:' in category from /Users/SudoPlz/Library/Developer/Xcode/DerivedData/App-fawgawfawfawfa/Build/Products/Debug-iphonesimulator/libRNSVG.a(RCTConvert+RNSVG.o) conflicts with same method from another category

I found that the same method names
that exist in
RCTConvert+RNSVG.h
exist in
RCTConvert+ART.h
Is there anything we can do about this? Some people want to use ART for animations as well as your library.

I have the same problem. iOS just crash when I use ART


Parse validation for git versions might be too lax

Over in https://phabricator.services.mozilla.com/D167275, an engineer submitted an entry that looks like this:
delta = "0.5.0 -> 0.5.0@git:a990c93#a990c93ec64eeab78f2292763d0715da9dba1d59"

I'm guessing it was an accident during hand-editing, but seems like cargo-vet should eagerly reject such entries from the local audit file.

This should be simple enough to fix - we'll want to check the length of the hash, and that every character is a lowercase hex digit, around here:
https://github.com/mozilla/cargo-vet/blob/7bf2a5e707a28f32e689778cd03d84861dc35105/src/format.rs#L70
TBH I don't know if cargo supports sha256 git repositories right now (I'm guessing not, because libgit2 doesn't seem to: https://github.com/libgit2/libgit2/discussions/5840), so we can probably hold off on supporting that until it becomes an issue.