Confirmed commit behavior is not RFC compliant

Confirmed commit behavior is not RFC compliant
Reference files
a-test.yang
module a-test {
    namespace "http://test.com/yang/";
    prefix test;
  container contain-1 {
    list data {
      key "name";
      leaf name {
        type string;
      }
      leaf value {
        type string;
      }
    }
  }
}

input1.xml
<contain-1 xmlns="http://test.com/yang/">
  <name>key</name>
  <value>first_value</value>
</contain-1>

input2.xml
<contain-1 xmlns="http://test.com/yang/">
  <name>key</name>
  <value>second_value</value>
</contain-1>

input3.xml
<contain-1 xmlns="http://test.com/yang/">
  <name>key</name>
  <value>third_value</value>
</contain-1>

Case 1 | Second confirmed commit before first times out
From RFC6241
If a follow-up
confirmed  operation is issued before the timer expires, the
timer is reset to the new value (600 seconds by default).  Both the
confirming commit and a follow-up confirmed  operation MAY
introduce additional changes to the configuration.

The second confirmed commit has to be treated as an update to the values of an existing confirmed commit.
It must not overwrite the backup values set by the original confirmed commit operation.
It seems every time a confirmed commit is received, the server creates a backup of the current running configuration. However, in this case, when the second commit is issued, no new backup must be done, since it is still the same transaction (the first confirmed commit was not confirmed by a confirming commit).
> edit-config --target candidate --config=input1.xml
> commit

> get-config --source candidate (at this point "key" has the value "first_value")
> get-config --source running (at this point "key" has the value "first_value")

> edit-config --target candidate --config=input2.xml
> commit --confirmed --confirm-timeout 600

> get-config --source candidate (at this point "key" has the value "second_value")
> get-config --source running (at this point "key" has the value "second_value")

> edit-config --target candidate --config=input3.xml
> commit --confirmed --confirm-timeout 10

> get-config --source candidate (at this point "key" has the value "third_value")
> get-config --source running (at this point "key" has the value "third_value")

wait 10 seconds...

> get-config --source candidate
> get-config --source running

Expected: both candidate and running have the value "first_value".
Actual: both candidate and running have the value "second_value".
Case 2 | Rollback is not happening for session terminated (confirmed commit without persist)
From RFC6241
If the session issuing the confirmed commit is terminated for any
reason before the confirm timeout expires, the server MUST restore
the configuration to its state before the confirmed commit was
issued, unless the confirmed commit also included a
element.

> edit-config --target candidate --config=input1.xml
> commit

> get-config --source candidate (at this point "key" has the value "first_value")
> get-config --source running (at this point "key" has the value "first_value")

> edit-config --target candidate --config=input2.xml
> commit --confirmed --confirm-timeout 600

> get-config --source candidate (at this point "key" has the value "second_value")
> get-config --source running (at this point "key" has the value "second_value")

> disconnect
> connect
> get-config --source candidate
> get-config --source running

Expected: both candidate and running have the value "first_value".
Actual: both candidate and running have the value "second_value".
The value will eventually go back to "first_value" when the timer expires but it should already been reverted since the connection was closed without issuing a confirming commit. The same result is obtained if you use, from another session, the command kill-session to kill the session that issued the confirmed commit.
Case 3 | Concurrent sessions issuing a confirmed commit are not resulting in error
From RFC6241
If the  element is not given in the confirmed commit
operation, any follow-up commit and the confirming commit MUST be
issued on the same session that issued the confirmed commit.
(But, this is also mentioned:)
For shared configurations, this feature can cause other configuration
changes (for example, via other NETCONF sessions) to be inadvertently
altered or removed, unless the configuration locking feature is used
(in other words, the lock is obtained before the
operation is started).  Therefore, it is strongly suggested that in
order to use this feature with shared configuration datastores,
configuration locking SHOULD also be used.

session 1
> edit-config --target candidate --config=input1.xml
> commit

> get-config --source candidate (at this point "key" has the value "first_value")
> get-config --source running (at this point "key" has the value "first_value")

> edit-config --target candidate --config=input2.xml
> commit --confirmed --confirm-timeout 600

> get-config --source candidate (at this point "key" has the value "second_value")
> get-config --source running (at this point "key" has the value "second_value")

session 2
> commit --confirmed --confirm-timeout 600

Expected: the session 2 confirmed commit must result in an error.
Actual: the session 2 confirmed commit works just as a normal confirmed commit.
Case 4 | Lock is preventing the rollback from happening
From RFC6241
A confirmed  operation MUST be reverted if a confirming
commit is not issued within the timeout period (by default 600
seconds = 10 minutes).

> lock --target running
> lock --target candidate

> edit-config --target candidate --config=input1.xml
> commit

> get-config --source candidate (at this point "key" has the value "first_value")
> get-config --source running (at this point "key" has the value "first_value")

> edit-config --target candidate --config=input2.xml
> commit --confirmed --confirm-timeout 5

> get-config --source candidate (at this point "key" has the value "second_value")
> get-config --source running (at this point "key" has the value "second_value")

wait 5 seconds...

> get-config --source candidate
> get-config --source running

Server messages:
...
[ERR]: SR: Module "a-test" is DS-locked by session 225.
[ERR]: NP: Failed restoring backup for module "a-test".

Expected: both candidate and running have the value "first_value".
Actual: both candidate and running have the value "second_value".

@michalvasko , do you agree these behaviors are not RFC compliant? Maybe my interpretation was not correct so that's why I'd like to confirm. Thank you

Everything was tested using the latest devel branch of the stack.

1), 2), 3) I agree that it was not working properly and it should now, all your use-cases added into tests.
4) I am not sure what to do about this case, I can now quote text about <lock>
      When the lock is acquired, the server MUST prevent any changes to
      the locked resource other than those requested by this session.
      SNMP and CLI requests to modify the resource MUST fail with an
      appropriate error.

We can talk about how to interpret "requested by this session" but I do not think commit timeout rollback is requested by any session. The previous configuration is actually still available and hence can theoretically be restored manually but will not be automatically. I think this is acceptable and is essentially a problem on the side of the user.

Thank you for the quick reply and fix @michalvasko.
Case 1, 2 and 3 all fixed now, retested with the latest changes.
Regarding case number 4, I raised the point you mentioned with my teammates and we understand when you say that the rollback does not belongs to any session. But by a user point of view, we believe that it makes more sense if the rollback indeed happens, as long as the rollback is due to a confirmed commit issued by the session which has the lock.
In the case a session acquires the lock while the confirmed commit timer was counting for an operation issued in another session, then we can expect the rollback to fail. Let us know your opinion considering this.

Let us know your opinion considering this.

I have already shared my opinion. I am not sure it makes sense to change locked data without the session requesting it (again left for interpretation). Also, there is an inconsistency when the change would be considered made by the session if it is still running and not if, for example, the server has crashed and is restoring the configuration after restart. If this change is so important to you, write to the NETCONF mailing list explaining the problem and I will implement whatever is agreed on.

You do not understand, such behavior is quite non-standard and constitutes practical challenges. Let's say the commit is being rolled back, it can take a lot of time with all sysrepo callbacks being called. What if during this time an edit-config comes on the session, what exactly should happen? Speaking specifically, sysrepo does not allow using its sessions in concurrently in separate threads (for good reason) so I would have to manually lock it, what else. Meaning the user may execute any RPC while the roll back is in process and would need to wait until it finishes.
I think commit rollback is a completely asynchronous operation and hence you cannot treat it as being performed by any session.

RFC 6241, § 8.4.1:
A confirmed  operation MUST be reverted if a confirming
commit is not issued within the timeout period (by default 600
seconds = 10 minutes).
....
For shared configurations ....
it is strongly suggested that in
order to use this feature with shared configuration datastores,
configuration locking SHOULD also be used.
Following this suggestion it is impossible utilise the timeout mechanism.

it is strongly suggested that in
order to use this feature with shared configuration datastores,
configuration locking SHOULD also be used.

When using locks I would expect you know exactly what is going to happen and for how long you are going to be holding the lock. So in the confirmed commit you specify some timeout just to be safe in case something goes wrong. But normally you either confirm or cancel the commit before the timeout and release the lock.

If taken literally, this is an impossible requirement. We are talking about rolling back potentially huge chunk of configuration and lots of things can fail for such an operation. A reasonable interpretation would be "confirmed operation MUST be attempted to be reverted" which would simply mean that no matter what happens, the roll back process must be initiated after the timeout elapses. That is all we have tried to implement, anyway.


Support for image prefix

This PR adds the ability to specify a prefix for images for addons, which defaults to pulling all images from StackHPC’s GitHub packages.
It also adds a CI job that uses skopeo to sync the required images, along with some small related reworking of the CI.

@scrungus
This is ready for review please. Thanks!


Updated extra.css

Updated the extra.css file to fit the frame to the edges of the page

Hold on merging please....

Okay
On Fri, 10 Jun 2022, 00:25 Tim Sutton, @.***> wrote:

Hold on merging please....
—
Reply to this email directly, view it on GitHub
https://github.com/kartoza/TheKartozaHandbook/pull/21#issuecomment-1151679138,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AUR3Q3ICFHCH7PNEHQOAOODVOJVOPANCNFSM5YKEWLEQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

@Mhlangabezi I am closing this in favour of us using the --with-pdf plugin rather...


Add a beta version of Windows 11 in React

Hello! Can you add a Insider Preview version of Windows 11 In React?
This version can have beta features.

We kind of already have this. New features are normally added to https://dev.blueedge.me before release.


Excuse me, I wanted to ask if there is support for textual inversion?,

The emphasis assignment works correctly for me, but I had doubts about whether it is possible to apply that emphasis in Textual Inversion (embeddings)

I haven't tried TI, but I think it should work

I currently preprocess the positive and negative prompts using.

prompt_text = pipeline.maybe_convert_prompt(prompt_text, tokenizer=pipeline.tokenizer)

This expands any single token that is associated with a TI into multiple tokens that the TI will understand.
you must use which ever tokenizer is going to be used for embedding generation with sd_embed.
Pass this converted prompt with the expanded tokens to sd_embed.
This works okay but is maybe not a perfect solution.
See here for a reference to what I am talking about: https://github.com/Teriks/dgenerate/blob/v4.3.0/dgenerate/promptweighters/sdembedpromptweighter.py

Hello @Teriks I just tried your solution, and it seems like the most appropriate way compared to others that I tried to apply. I did some testing and I liked the results, I'm still testing with style type embeddings. Thank you very much for comparing your solution, and another thank you because for your "dgenerate" repository, I found this incredible library, also thanks to the author of the library @xhinker, it would be incredible if you continue to share this library, I love it.


[DOCS]Improve Install Google Chrome Beta section

The with should be applied to the uses, not run. Also, some minor spelling checks :)

@ueokande do you mind checking it out?


Please add some example with real Dataset, it is not working.

Please , use some real dataset ( Ex : which are avaible Kaggle, House pricing .etc..)
we are getting problem in giving features and target tata.
Catboost is giving best results in python.
Please fix it in julia and add some examples with hyper parameter tuning.
Please help,
Thank you

Is there an example or tutorial you like for python? This is just a light wrapper of the python version so any examples there should translate pretty easily. If you link one you are having trouble translating I could help.
If you are getting errors, please post them as minimal examples with the error message and stack trace. It could be a bug in the wrapper code here.

I have dataFrame data, Pylist is not working.
https://github.com/catboost/tutorials/tree/master/regression
Please add any example with these features (https://github.com/catboost/tutorials/tree/master/regression)
input feature data, lable data , is hard to define.
Please, add one example.
Catboost is best, nothing comparable to catboost.
Please provide good wrapper in julia with exmaple.
Thank you.

I have dataFrame data, Pylist is not working.

If you want to work with DataFrames.jl/Tables.jl, you have to convert it to a python table using the pytable function in PythonCall.jl.
https://cjdoris.github.io/PythonCall.jl/stable/compat/#PythonCall.pytable
Here is an example:
using CatBoost
using DataFrames
using PythonCall

df_train = DataFrame(a=[1,4,30], b=[4,5,40], c=[5,6,50], d=[6,7,60])
train_data = pytable(df_train)
df_eval = DataFrame(a=[2,1], b=[4,4], c=[6,50], d=[8,60])
eval_data = pytable(df_eval)
train_labels = PyList([10, 20, 30])

# Initialize CatBoostRegressor
model = CatBoostRegressor(; iterations=2, learning_rate=1, depth=2)

# Fit model
fit!(model, train_data, train_labels)

# Get predictions
preds = predict(model, eval_data)

Thank you . it is working.
Could you please tell me how to use Pool function and cv function ?

Any ideas?

I can not replicate this. What platform are you on? Can you try in a fresh environment with just CatBoost, DataFrames, PythonCall? Does the test suite pass for you?

Running a fresh julia file in VScode I get this error when running CatBoost (the error doesn't show up in Jupyter Notebooks):

Thanks @tylerjthomas9  this was a huge help! It now works. Maybe we should add some of these parts to the landing page on how to get this up and running and how to use native dataframes. I know there are many like myself and @MrBenzWorld who really want to use Julia but documentation is either outdated or missing altogether.
Just my $0.02

how to use native dataframes

What do you mean by native dataframes? Pandas or DataFrames.jl?
I will look into adding more documentation, especially on getting started and troubleshooting PythonCall.jl issues. If you come up with other things, feel free to open another issue where we can track potential documentation deficiencies in one place.

how to use native dataframes

What do you mean by native dataframes? Pandas or DataFrames.jl?
I will look into adding more documentation, especially on getting started and troubleshooting PythonCall.jl issues. If you come up with other things, feel free to open another issue where we can track potential documentation deficiencies in one place.

Thanks for the added documentation & sorry for not being concise - I meant DataFrames.jl - it would be great to see how this package works from start to finish including getting plots up and running (I have, after many hours, gotten everything to work except plots = true despite having all of the necessary libraries installed).


certain plugins’ testsuites are now broken

@ivanteoh?  (those are all yours: book_figure, link_figure, metadata)

I fixed these.

@ralsina thanks. I need to find a way to track the issue that I responsible.


Update sbt-java-formatter to 0.8.0

Updates com.lightbend.sbt:sbt-java-formatter from 0.6.1 to 0.8.0.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.lightbend.sbt", artifactId = "sbt-java-formatter" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "com.lightbend.sbt", artifactId = "sbt-java-formatter" }
}]

labels: sbt-plugin-update, early-semver-major, semver-spec-minor, commit-count:n:3

Codecov Report
Base: 59.87% // Head: 59.87% // No change to project coverage :thumbsup:

Coverage data is based on head (d4f4b53) compared to base (df19298).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #417   +/-   ##
=======================================
  Coverage   59.87%   59.87%
=======================================
  Files          30       30
  Lines         987      987
  Branches       64       73    +9
=======================================
  Hits          591      591
  Misses        396      396

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


when npm run build : events.js:154  throw er; // Unhandled 'error' event

I have cloned draft-js and after npm i, at the first attemp of npm run build , it came across the error below :
Faxra-mac:draft-js abdi.fakhruddin$ sudo npm run build

><EMAIL_ADDRESS>build /Users/fakhruddinabdi/WebstormProjects/archive/draft-js
> gulp

[10:57:57] Using gulpfile ~/WebstormProjects/archive/draft-js/gulpfile.js
[10:57:57] Starting 'default'...
[10:57:57] Starting 'check-dependencies'...
[10:57:59] Finished 'check-dependencies' after 2.83 s
[10:57:59] Starting 'clean'...
[10:57:59] Finished 'clean' after 4.61 ms
[10:57:59] Starting 'modules'...

events.js:154
     throw er; // Unhandled 'error' event
     ^
ReferenceError: [BABEL] /Users/fakhruddinabdi/WebstormProjects/archive/draft-js/src/Draft.js: Unknown option: /Users/fakhruddinabdi/WebstormProjects/.babelrc.presets
   at Logger.error (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/babel-core/lib/transformation/file/logger.js:58:11)
   at OptionManager.mergeOptions (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/babel-core/lib/transformation/file/options/option-manager.js:126:29)
   at OptionManager.addConfig (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/babel-core/lib/transformation/file/options/option-manager.js:107:10)
   at OptionManager.findConfigs (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/babel-core/lib/transformation/file/options/option-manager.js:168:35)
   at OptionManager.init (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/babel-core/lib/transformation/file/options/option-manager.js:229:12)
   at File.initOptions (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/babel-core/lib/transformation/file/index.js:147:75)
   at new File (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/babel-core/lib/transformation/file/index.js:137:22)
   at Pipeline.transform (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/babel-core/lib/transformation/pipeline.js:164:16)
   at DestroyableTransform._transform (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/gulp-babel/index.js:30:20)
   at DestroyableTransform.Transform._read (/Users/fakhruddinabdi/WebstormProjects/archive/draft-js/node_modules/readable-stream/lib/_stream_transform.js:159:10)

npm ERR! Darwin 15.3.0
npm ERR! argv "/usr/local/Cellar/node/5.7.0/bin/node" "/usr/local/bin/npm" "run" "build"
npm ERR! node v5.6.0
npm ERR! npm  v3.6.0
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>build: `gulp`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build script 'gulp'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the draft-js package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     gulp
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs draft-js
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls draft-js
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/fakhruddinabdi/WebstormProjects/archive/draft-js/npm-debug.log

#https://github.com/babel/babel-loader/issues/132

https://github.com/babel/babel-loader/issues/132


clippy warns with clippy::needless_update when using the git_testament! macro

Hi, I saw this library in the latest release of rustup, and it's much nicer than the typical build.rs method!
I noticed when adding it to my program, clippy gives the clippy::needless_update warning when using the git_testament! macro. This is with git-testament 0.1.4 and git-testament-derive 0.1.5.
$ rustc --version
rustc 1.34.1 (fc50f328b 2019-04-24)
$ cargo --version
cargo 1.34.0 (6789d8a0a 2019-04-01)
$ cargo clippy --version
clippy 0.0.212 (726176e3 2019-04-18)
$ cargo clippy
   Compiling foo v0.1.0 (/home/magicmaaaaan/foo)
warning: struct update has no effect, all the fields in the struct have already been specified
  --> src/main.rs:11:1
   |
11 | git_testament!(TESTAMENT);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: #[warn(clippy::needless_update)] on by default
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_update

    Finished dev [unoptimized + debuginfo] target(s) in 2.76s

Hi,
Thanks for taking the time to look at git_testament and to file this issue.  The struct initialiser is used in an attempt to be more forward-compatible and to reduce the exact-tie between git_testament and git_testament_derive -- As such, I think the right answer is to silence that clippy lint rather than to remove it.  I shall sort out a patch so that the next release of git_testament_derive has #[allow(clippy::needless_update)] on the requisite item.  In the meantime, you can probably silence that by putting it on the git_testament!(TESTAMENT); statement if you want.
Thanks again,
D.


Suggestion: Add option to change default description and version

Hello, sorry if this is not the right way to suggest a feature. (I'm not very familiar with Github.)
I wanted to suggest adding an option to let users change the default description and version. Right now, the description text defaults to "My Electron application description" and the version number defaults to "1.0.0", but having the option to change this could be useful.

Awesome, thanks for this!
The project was supposed to pull in the version number from GitHub (by default, each new build increases the version number by 1) but that does seem to be broken.
I'm merging this in as-is for now, since it's better than what we have. I'm not sure the version number always being the auto-incrementing one is the right choice, so I'll want to think about the best way to support both.


new(pkg/build): only attempt to build supported drivers.

What type of PR is this?
/kind feature
Any specific area of the project related to this PR?
/area pkg
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

@FedeDP: The label(s) kind/feature, area/pkg cannot be applied, because the repository doesn't have them.

In response to this:

What type of PR is this?
/kind feature
Any specific area of the project related to this PR?
/area pkg
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
NONE

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

There is not a label identifying the kind of this PR.
Please specify it either using /kind <group> or manually from the side menu.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This is not really needed since driverkit would tell us immediately that the required driver is unsupported.
/close

@FedeDP: Closed this PR.

In response to this:

This is not really needed since driverkit would tell us immediately that the required driver is unsupported.
/close

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Hotfix mw data getting wrapper tests

tests didn't pass because of some instance methods that should have been class side and a call of the parser with nil

(the commits from the MWParser are shown here, because github thinks, they weren't merged yet)


Fixed email Validation bug, code refactoring and other cosmetic changes.

Submitting under hacktoberfest-2022. Thank you.

Remember user's selected theme, show angry and smiling emoji etc etc....

@fineanmol


Note the integration from magisk module

https://github.com/Goooler/systemless-fcm-hosts

CC @entr0pia


greetings action broken

The issue is caused by https://github.com/actions/first-interaction/issues/23

I will temporary disable greeting workflow


Add spoc run command

What type of PR is this?
/kind feature
What this PR does / why we need it:
This allows to run a seccomp profile for a custom command or binary.
Which issue(s) this PR fixes:

Refers to #1482
Does this PR have test?

Yes
Special notes for your reviewer:
None
Does this PR introduce a user-facing change?

Added `spoc run` command for experimental testing of seccomp profiles.

@kubernetes-sigs/security-profiles-operator-maintainers PTAL

/lgtm cancel
Let's wait for another maintainer to review.

Great idea and implementation! Who needs ChatGPT when @saschagrunert is around! :mechanical_arm: :rocket:
Adding a hold in case other folks may want to comment before the merge.
/lgtm
/hold

Awesome, yeah we can follow-up on the other topics since I'm also planning to push more docs and testing around the features. I just updated the PR to propagate the correct exit code via spoc run.

/unhold
Thanks for all the reviews!


[release-1.21] Update embedded kube-router network policy controller

Backport kube-router network policy controller update from #3373

Not a blocker for release. this is a nice-to-have soon, ideally before 1.22 release.

Work should land on #3373 first

Validated using 1.21.2-rc1+k3s2

There are many new rules added, including the one we are looking for. See https://github.com/k3s-io/k3s/issues/3373#issuecomment-877527072
Standard checks continue to pass after running these steps


Certificate Chain Support

Hi guys,
I'm having this issue in that I need to have the certificate chain built in to the load balancer. I don't see a param in emp ssl-cert-add. When I manually add the certs to the ELB all is well, but then the empire doesn't allow me to do any deployment to the app because the ELB doesn't match the way it was initially configured. Confused as to how to set this up properly or if intermediary cert chain is not supported in empire.

Short Answer: I don't believe this is supported right now.
Long Answer: I think you could initially use emp to setup the app and load balancer, then use the AWS CLI to upload the correct cert and certificate chain using the same certificate name. I'm actually not sure if IAM let's you change an existing cert or if you have to delete it first, but as long as it's the same name as the one that was uploaded via emp, it should work. The cert name within IAM should be the app id, which you can find from emp info -a <app>.
Let me know if that works out for you!

You should just include the whole certificate chain in the .crt along with
your certificate. We had the same problem, included the intermediate
certificate in the same CRT file that had only our issued certificate.
Works like the charm and emp will happily create the ELB correctly.
That being said, we've bumped into IAM limits even though our limit is
double the default (40 certificates per account). Any chance to have hashed
certificate names implemented so they get reused?
On Tue, Sep 15, 2015 at 10:57 PM Eric Holmes<EMAIL_ADDRESS>wrote:

Short Answer: I don't believe this is supported right now.
Long Answer: I think you could initially use emp to setup the app and
load balancer, then use the AWS CLI to upload the correct cert and
certificate chain using the same certificate name. I'm actually not sure if
IAM let's you change an existing cert or if you have to delete it first,
but as long as it's the same name as the one that was uploaded via emp, it
should work. The cert name within IAM should be the app id
https://github.com/remind101/empire/blob/master/ssl.go#L77, which you
can find from emp info -a .
Let me know if that works out for you!
—
Reply to this email directly or view it on GitHub
https://github.com/remind101/empire/issues/654#issuecomment-140536990.

^^ what @rgabo said.  Have you tried including the certificate chain in your certificate @kamkudla?  That should work/be supported.

@ejholmes Seems like emp info is broken for me I get error: Unexpected error: 404 Not Found for all apps even ones that are live and freshly set up. I was only able to get the certificate name via ELB AWS page. Didn't get around to trying to overwrite the cert. I went with what @rgabo suggested and it worked like a charm. Works great, thanks.

We should most definitely document this in the setup in the meantime while we extend the method to accompany the certificate chain in the upload

@kamkudla emp info should work - it's working in our environment.  What version of empire are you running/how did you set it up?

Yeah, it seems like emp is just not finding the API endpoint, thus returning a 404. Updating empire should fix it :).

@phobologic @benguillet Looks like I'm using version 0.9.1.
I pulled it with `go get -u github.com/remind101/emp'.

Sorry - I mean what version of empire?  How did you create your empire environment?  This isn't likely an emp CLI problem.

@phobologic stacker was used to set up the stacks. Regarding empire, not sure if the latest empire env was used. How could I check? Through empire API?

Cool - can you give me the mappings: section from the empire.yaml that you used?  Or, alternatively, if you could give me the AMI that your controllers are using that'd work too.
Also the region you are running empire in, thanks!
Updating Empire with stacker usually means doing a stacker update with a new AMI, then killing each controller/minion host one at a time, allowing Autoscaling to update it.  I can give you more info about that if it turns out you need to update.

Sure thing. Here's the mappings: from from my empire.yml
  AmiMap:
    us-east-1:
      NAT: ami-c02b04a8
      ubuntu1404: ami-c135f3aa
      empire: ami-3186585a
    us-west-1:
      NAT: ami-67a54423
      ubuntu1404: ami-bf3dccfb
      empire: ami-2fd5286b
    us-west-2:
      NAT: ami-75ae8245
      ubuntu1404: ami-f15b5dc1
      empire: ami-85ede7b5
      ecs: ami-abf4fd9b

Controllers are using ami-85ede7b5 and the minions are using ami-abf4fd9b which is a custom image that contains logsprout and datadog integrated

Any ideas, guys?

Sorry, haven't had a chance to look at this today - I'll try to get some time in a bit.

Thanks @phobologic. Looking forward. I do believe I may be running an old image.

Hmm - so ami-85ede7b5 is the latest AMI in us-west-2... and yeah, emp info came later!  Can you go ahead and open a bug/request in empire_ami to get a new one spun?  I'll make that happen ASAP - I need to revisit the current code in that repo anyway to see if I can get docker 1.8.1 running or if I should roll back to 1.7.0 entirely (there is a weird networking bug in 1.8.1 that I found after starting to roll it out).
I'm going to go ahead and close this bug for now so we can continue any conversation in the right place.  Thanks for the info @kamkudla


update nan to 1.8.4

Installing this package fails on io.js because io.js uses a more recent version of v8 which changes Signature::New. (seen on nan issue 308).
Updating nan to 1.8.4 fixes this issue, and npm test runs fine.
Tested on Node 0.12.2 and io.js 2.0.1 .

Coverage remained the same at 98.4% when pulling 58d632b49312fe342d799b794b4d0158f0cb16cb on th507:master into e742a41649abf90bbaa225a7039063c24bc75a2a on EyalAr:master.

Closing this pull request in favor of #142.


Feature needed : File Import

Hello,
Can you plan to add the feature to add a file to an asset ?
Il will be great to do this.

That would be nice and useful feature. Unfortunately Snipe It api (https://snipe-it.readme.io/reference) lacks support for file operations.  There's at least one request for files api snipe/snipe-it#9413   .So before SnipeitPS can have this, it needs api to work with.

Image upload support is there already. So after next Snipeit release you can set images for assets, accessories etc...

The support to upload files is now avaliable.
https://github.com/snipe/snipe-it/pull/15362


broken link

the link in this line is broken
The README.md file describes how to change the hardware target for the samples.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d195d27b-e2ea-6583-7d6d-b33c8392c6bf
Version Independent ID: 59e2ee21-3862-58f0-be3a-16313d0e1251
Content: Release Notes 19.05
Content Source: AzureSphere/articles/resources/release-notes-1905.md
Product: azure-sphere
GitHub Login: @pennyo
Microsoft Alias: azurespheredocs

@ozhernan Thanks for the feedback! We are currently investigating and will update you shortly.

@pennyo Would you please review the PR with the url change? Thanks!

@ozhernan Closing this issue since the fix has been merged.


ion-tabs: select() doesn't immediately set active state on initialized ion-tab

Ionic version:  (check one with "x")
[ ] 1.x
[x] 2.x
I'm submitting a ...  (check one with "x")
[x] bug report
[ ] feature request
[ ] support request => Please do not submit support requests here, use one of these channels: https://forum.ionicframework.com/ or http://ionicworldwide.herokuapp.com/
Current behavior:
When a tab that's already been initialized (clicked) is selected again programmatically with Tabs::select() the tab content is rendered but the tab button itself is not active until some other action repaints the view.
Note that if switching to a tab for the first time this works as it should.
Expected behavior:
The tab button should gain active state immediately.
Steps to reproduce:
Scenario: 2 tabs, index 0-1

Open app, tab 0 is active by default
Click tab 1
Click tab 0
Call Tabs::select(1)
The contents of tab 1 is now rendered, but the tab itself remains inactive until something triggers  a repaint. The ionChange() event is called as it should.

Related code:
ionic start --v2 myApp tabs
tabs.html: add tab ref: <ion-tabs #tabs ..
tabs.ts: declare tab ref: @ViewChild('tabs') tabRef: Tabs;
tabs.ts: declare helper function in constructor: (<any>window).switchTab = (index: number) => this.tabRef.select(index);
call this function from browser console in reproduction step 4: switchTab(1);
Other information:
Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
Cordova CLI: 6.5.0
Ionic Framework Version: 2.3.0
Ionic CLI Version: 2.2.1
Ionic App Lib Version: 2.2.0
Ionic App Scripts Version: 1.1.4
ios-deploy version: 1.9.0
ios-sim version: 5.0.13
OS: macOS Sierra
Node Version: v6.9.2
Xcode version: Xcode 8.3 Build version 8E162

A workaround is to call ApplicationRef.tick() after the select()

Same bug for me.

I stumbled upon this issue as well.
I have a button on a side menu, that should navigate to a page, that has to be a child page of a tab.
I think tabsRef.select() should be an Observable.
Calling tabsRef.getSelected().push() right after does not behave as expected. This is because tabsRef.getSelected() holds the previous tab at this point, and not the new one yet.
For now a have a dirty workaround with setTimeout.

@dima-gusyatiner  Im with the same problem here. Timeout works like you told. Any news about how to solve this?

@metz0r solution worked, Sep 2018

@metz0r can you show some snippet on how to apply on it? thanks best regards

Hi, is there a solution for this problem ? I tried the timeout solution and ApplicationRef.tick() also, but it didn't work (ionic v3.9.2).
Thanks.


Add custom env file path

Loads the "env" file from a custom path if your file containing environment variables is located elsewhere.

added a new commandline argument, envPathArg.
Add the config to the dotenv config only if it has been set, otherwise don't change anything. (link to dotenv path option)

@goce-cz can you please take a look at this?

I am not authorized to merge, so if you can help me do it, that'd be great! 🙏


Restrict products

Task: Restrict products
High Level Description
Only allow creating options products which are fully collateralized which won't require any liquidation mechanism.
Specific Changes
Added a require statement in whitelist to check that only put option products where the strike == collateral are allowed and call options where underlying == collateral are allowed.
Code

[x] Unit test 100% coverage
[x] Does your code follow the naming and code documentation guidelines?

Documentation

[ ] Is your code up to date with the spec?
[ ] Have you added your tests to the testing doc?

Plz add tests for this line!

Plz add tests for this line!

This will break the integration tests in Yield Farming: Naked Put Option closed before expiry flow, do u want to just ignore those for now by adding xit

This will break the integration tests in Yield Farming: Naked Put Option closed before expiry flow, do u want to just ignore those for now by adding xit


Associate insights app template

Docs Build status updates of commit 47b5751:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/assets/images/associate-insights-app.PNG
:white_check_mark:Succeeded
View

msteams-platform/assets/images/associate-insights-app2.png
:white_check_mark:Succeeded
View

msteams-platform/samples/app-templates.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Why is Flow implemented in OCaml?

Notice: Please don't take this Issue as offensive or trolling, its a genuine doubt.

So, is there any special reason why Flow is made mostly upon OCaml instead of JavaScript? I'm sure there are definitely some reasons to choose using it (probably performance, if I had to guess), but as I couldn't find anything on Facebook's Dev Blog or Flow's documentation, I felt the need to ask over here.
Are there any special reasons to do so, instead of developing it fully in JavaScript (and enabling its installation via npm on all OSes)?
Great project, and badass work on here thought, keep the good job and thanks for this tool!

And what do we use to do static analysis for flow that is implemented in javascript? it's a chicken-egg problem

Ocaml being a Statically typed functionnal language, is simply a better tool for this kind of job.

@arypurnomoz, like @kazzkiq, I do not mean to be offensive but isn't bootstrapping common? I know Flow isn't a compiler per se, but I think the same principles apply. I'm only writing this because I'm curious to be proven wrong.

@arypurnomoz I cant even start to imagine the quirks both TypeScript and Flow core teams had to deal with when building those tools, and they surely had to face obscure bugs and architecture concerns, but TypeScript is made 100% in ... TypeScript. So if it managed to use its own language to make a compiler, why Flow didn't followed the same path? I just ask that because I got curious about Flow choosing OCaml instead of JavaScript or Flow's syntax itself.

A lot of people including the Flow team believe that functional programming
languages like OCaml are great for writing compilers. These kinds of
programs make heavy use of abstract data types and pattern matching. Plus,
OCaml's nice interop with C and fast running times make it an ideal
language for systems programming / scaling for perf.
Bootstrapping is cool but we'd still have a compiler to maintain after the
initial coolness fades away.
What's more, OCaml can be compiled to JS so we don't have to give up on
JS interop either.
-Avik.
On Saturday, April 2, 2016, Claudio Holanda<EMAIL_ADDRESS>wrote:

@arypurnomoz https://github.com/arypurnomoz I cant even start to
imagine the quirks both TypeScript and Flow core teams had to deal with
when building those tools, and they surely had to face obscure bugs and
architecture concerns, but TypeScript is made 100% in ... TypeScript. So if
it managed to use its own language to make a compiler, why Flow didn't
followed the same path? I just ask that because I got curious about Flow
choosing OCaml instead of JavaScript or Flow's syntax itself.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/facebook/flow/issues/1619#issuecomment-204806691

@avikchaudhuri Great, thanks!


UI / UX Review

Requirement:
Review application for usability and experience problems and make recommendations on how to fix them.
Acceptance Criteria:
Documented list of issues
Review list with Principal SWE
Create issue tickets for backlog as needed

Waiting for Mike and Laura to review.

Completed Review and turned in writeup.


Calling reload_schedule! resets pending waiting every jobs

It seems that whenever you call Sidekiq::Scheduler.reload_schedule!, it resets all jobs scheduled for every.
This means that if you call Sidekiq::Scheduler.reload_schedule! once per hour, but you have a job that's scheduled to run every two hours (or longer), the job will never be started.

Hi,
Are you modifying the schedule by calling Sidekiq.set_schedule ?
In that case calling Sidekiq::Scheduler.update_schedule instead, will work. (I'll modify the README in case you confirm)
Are you calling reload!_schedule from a sidekiq job?

I'm modifying the schedule from a sidekiq job, yes:
  def call
    Sidekiq.schedule = generated_schedule
    Sidekiq::Scheduler.reload_schedule!
    del('*', 100, false)
  end

I'm wondering if the del is getting in the way, I'll try remove that and see if that resolves the issue.

@snmgian I just ran into this issue when trying to work with the API. I think that updating the documentation would help a lot.


Add KNN and ROC curve

The following pull request adds, fixes, and/or modifies:

Adds a simple KNN classifier (needed for the following point)
Adds the ROC AUC metric (solving #32)

I performed the following prior to filing this pull request:

[x] Tested that my change does not break the analysis pipeline
[x] Ran a linter through my code
[x] Update environment dependencies if my code introduces a new package

Looks great. KNN is a great addition to the research. But was wondering a few things:

In you README file you refer to results/all_scores.csv but this file is missing from the git
Also what is it you exactly mean with '0.300 ± 0.010, 10, 15, 25, 35 or 50. ' in the README?

I try to run your code at the moment but this might take some time (partly because I run everything in Windows Subsystem Linux, partly other thing to do on my days off :) )

Small thing for the next time: split your work in smallest pieces. You have now KNN researched (second pull request) and the functionelity of ROC AUC (first pull request). Makes it easier for use to check, we can approve ROC AUC and add it even is KNN has some work to do

Thanks! Just added the results and tried to clarify that. KNN is deterministic so I ran it with 6 different values for the number of neighbors considered to estimate the robustness of the performance. The proper way to do that would be with a proper cross-validation.
You're right, the issue was that the ROC curves needed the KNN (or any probabilistic prediction model) to be tested so I thought bundling them in a single PR was the least inconvenient solution for now. KNN implementation does not need ROC curves so we can probably split it if this is needed.
Cheers.

Cool, it means you have parameter estimation in your solution. But with deterministic solutions it becomes a bit more tricky, I am having the same problem with support vector machine (SVM). Common way of solving it is with cross-validation  but that will be out of scope for this pull request.
I will test if I can run your code now

@wagenrace @koalive do you need me to do anything here? I can try running on my local machine

@koalive sorry for the late reaction. I missed your updates. Everything look good, I will merge it to the project


Doesn't work if yabai is not installed in /usr/local/bin

I use Nix to manage my MacBook, and it doesn't install things to /usr/local/bin. It would be awesome if there was a way to inject the yabai path into the scripts rather than it being hard-coded to /usr/local/bin/yabai, even if that remains the default.

I pushed this modification, tell me if it is working for you or not.

OK, I'll try it out. I may have to modify the application path so that Ubersicht gets the right paths for that approach to work, but I can do that.
For anyone else finding this, the command on your mac to fix the paths so that yabai can be found is:
sudo launchctl config user path /path/to/yabai/bin/directory:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

That will add the new directory to the list that non-shell-based GUI apps search.

Updating the path for user apps didn't work - Ubersicht still can't see the path I need, or there's some other issue. Manually editing the yabai calls in lib/yabai.js and the shell scripts to point to it's location on my machine does work. I'm OK with saying that I'm in the minority of users, and I'll modify my behavior to use this if I want to. The change for this issue should likely be rolled back if it works for the majority of users.

OK, I've fixed it with the current change. Even though Ubersicht is a user-land app, the run function is using the system level app path. Updating it to contain the yabai path works.
So, the proper command to add the non-standard yabai path for this change would be:
sudo launchctl config system path /path/to/yabai/bin/directory:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

I'm still thinking that allowing the yabai binary path to be changed via the simple-bar settings might be more user-friendly.

Thanks for you research, I think I'll implement a setting which will permit user to override default yabai path as you suggested it.

The latest version of simple-bar includes a setting which allow the user to set his own custom path:

Just tried this out, works perfectly. Thanks!


Get track start time (cue)

This one for example, the second track starts from 5:55, it's a very important info for a track list.
btw could it get the name of the set for now? It's Miss Melera - Colourizon 101 February 2021-02-17 for this one.

Sounds like an interesting feature. I'll get back to this when I have time.
The name of the set is accessable via the "title" attribute.

@dashezup The pull request above (#5) should have what you're looking for.

for num, cue, track in zip(tracklist.track_nums, tracklist.cues, tracklist.tracks):
    print(f"{num}. {cue} {track.title}")

thanks, just checked it works.


Multiple select

I can’t achieve filtering with a multiple select. The filtering callback returns no object.
It works fine with a simple one with :
FJS.addCriteria({field: ‘myField’, ele: ‘#myselect’, selector: ‘select’ })
It works fine with checkbox instead of multiple select.
Is this a limitation or am I missing something?
in master branch (not 2.0 which I do not test)

I am also seeing this issue when trying:
HTML:
<select class="form-control" id="degree_filter">         <option value="all">All</option>         <option value="Ph.D.">Ph.D.</option>         <option value="M.D. / Ph.D.">M.D. / Ph.D.</option>         <option value="M.S.">M.S.</option>   </select>
JS:
FJS.addCriteria({field: 'related_degrees_filterable', ele: '#degree_filter', all: 'all'});
JSON:
"related_degrees_filterable": [         "M.D. / Ph.D.",         "M.S."       ],
I receive the following error: filter.js:327 Uncaught TypeError: qFn is not a function
Any insight appreciated!

For multiple select only single value in JSON will work.
i.e
"related_degrees_filterable":"M.D"

For array need to change code in selection and filtering logic.

Thanks Jiren!


Output type inference

Addresses an ask from #10354
As a companion to #10418, this PR updates output type inference to take the supplied output value into account.
One potential drawback of inferring more detailed type information from the value is that while Bicep can easily figure out an extremely narrow type for a statement written in Bicep, the compiler isn't as smart when looking at JSON templates. E.g., the type of foo would be 'foo' in output foo string = 'foo' and would just be string in:
{
  ...
  "outputs": {
    "foo": {
      "type": "string",
      "value": "foo"
    }
  }
}

The above limitation would mean that type inference would work for a local bicep module but not one that had been published to the registry.
Microsoft Reviewers: Open in CodeFlow

Per a team discussion, publishing a Bicep module to a registry should not be a lossy proposition. Inferred type information would need to be persisted in the compiled JSON template.


Page slowing on http://blackhawks.nhl.com/club/page.htm?id=106807

Hobson, Jessen L
3:32 AM (4 hours ago)
to me
Significant slowing on http://blackhawks.nhl.com/club/page.htm?id=106807
The screen would shrink and then go back to normal size and then repeat that. Did work eventually (30 seconds)

Multiple events for "did-frame-finish-load" are being fired in certain pages including this page. As a result, screenshots are taken simultaneously for each "did-frame-finish-load" event.
Need to see if there are ways to check for duplicate "did-frame-finish-load" events on certain pages

Issue fixed: 60eff48f05ae90099c69ed770831ba5d33d9e80d


docs: add table of contents to README.md

Nice. Is this generated by hand or automatically?

Manually for now.
I almost didn't add a table of contents because I found this feature, but the fact that I wasn't already aware of this feature, was a mark against relying it.


MaterialManager refactor

Hi all,
While working on the font antialiasing, I realized that there is an problem with MaterialManager.
@hybridherbst proposed a performance improvement here that require precomputing a value to set as uniform. But where should this precomputing take place ? In _makeTextMaterial ? In getFontMaterial ? in updateTextMaterial ?
This is the issue that @Enijar faced when implementing variable border radius, they had to duplicate code in three functions of the module.
To fix this issue I propose that this.backgroundUniforms and this.textUniforms be set in MaterialManager constructor, and only one function should be in charge of calling this.getFontOpacity() and such to update this.textUniforms.

To fix this issue I propose that this.backgroundUniforms and this.textUniforms be set in MaterialManager constructor, and only one function should be in charge of calling this.getFontOpacity() and such to update this.textUniforms.

Is the idea of this issue to explicitely focus on that proposal or / also to open a discussion about a wider refactor ?

It's really a problem with MaterialManager here, but feel free to open another issue to propose a more general refactor.


Merge SQL failing with ParseException

Hi Team,
I am trying to perform MERGE on HiveAcid talbes using Qubole Spark-ACID, but was facing below errors:
Created an assembly jar from latest code from mater and tried to execute MERGE statement using spark.sql from spark-shell:
/spark-2.4.3-bin-hadoop2.7/bin/spark-shell --jars /sandbox/spark-acid-assembly-0.5.0.jar --conf spark.sql.extensions=com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension --conf "spark.hadoop.yarn.timeline-service.enabled=false"
scala> spark.sql("MERGE INTO TARGET_TABLE AS T  USING SOURCE_TABLE AS S ON t.key_col= s.key_col WHEN MATCHED THEN UPDATE SET                    col_1 = s.col_1, col_2 = s.col_2 WHEN NOT MATCHED THEN INSERT VALUES ( s.key_col, s.col_1, s.col_2)")

org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'MERGE' expecting {'(', 'SELECT', 'FROM', 'ADD', 'DESC', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'INSERT', 'DELETE', 'DESCRIBE', 'EXPLAIN', 'SHOW', 'USE', 'DROP', 'ALTER', 'MAP', 'SET', 'RESET', 'START', 'COMMIT', 'ROLLBACK', 'REDUCE', 'REFRESH', 'CLEAR', 'CACHE', 'UNCACHE', 'DFS', 'TRUNCATE', 'ANALYZE', 'LIST', 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT', 'LOAD'}(line 1, pos 0)

== SQL ==
MERGE INTO TARGET_TABLE AS T  USING SOURCE_TABLE AS S ON t.key_col= s.key_col WHEN MATCHED THEN UPDATE SET                    col_1 = s.col_1, col_2 = s.col_2 WHEN NOT MATCHED THEN INSERT VALUES ( s.key_col, s.col_1, s.col_2)
^^^

  at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:241)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
  at com.qubole.spark.datasources.hiveacid.sql.execution.SparkAcidSqlParser.parsePlan(SparkAcidSqlParser.scala:56)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
  ... 49 elided

@srinikvv how was this created: /sandbox/spark-acid-assembly-0.5.0.jar ?

@amoghmargoor I

@srinikvv how was this created: /sandbox/spark-acid-assembly-0.5.0.jar ?

I built this assembly jar with latest code from master branch

Can you check with this jar once: https://drive.google.com/file/d/1sqsFzUtyrWvXfE7g_Q8brNHqMnb14Gvv/view?usp=sharing ? And also share your jar please, if the one i provided works.

@amoghmargoor Tried the jar provided and still see the same issue.

@srinikvv this is working fine, i rechecked. This would happen only if sql extension is not getting added on your end. It is difficult for me to figure out why that would not happen. But Can you check that angle by

putting log messages in https://github.com/qubole/spark-acid/blob/master/src/main/scala/com/qubole/spark/hiveacid/HiveAcidAutoConvert.scala where rule new parser is injected and new rule too.

Checking if UPDATE or DELETE is working or not because we have extended grammar for them too.

@amoghmargoor I checked UPDATE is working fine. Also the error stack trace indicates an exception at:
https://github.com/qubole/spark-acid/blob/a36f56d2030bba59cf5c1735688234555d70bf83/src/main/scala/com/qubole/spark/datasources/hiveacid/sql/execution/SparkAcidSqlParser.scala#L56
Hence I believe this is not an issue with sql extension not getting added. Please check and advice

@srinikvv Can you check what is the token we get here: https://github.com/qubole/spark-acid/blob/a36f56d2030bba59cf5c1735688234555d70bf83/src/main/scala/com/qubole/spark/datasources/hiveacid/sql/execution/SparkAcidSqlParser.scala#L112 ? and is that function returning true or false ?

@srinikvv this stack trace you printed is not corresponding to current code too:
  at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:241)   at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117)   at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)   at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)   at com.qubole.spark.datasources.hiveacid.sql.execution.SparkAcidSqlParser.parsePlan(SparkAcidSqlParser.scala:56)
Line 56 doesn't have function call to parsePlan. Please recheck your jars.

@srinikvv this stack trace you printed is not corresponding to current code too:
 at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:241) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117) at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48) at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69) at com.qubole.spark.datasources.hiveacid.sql.execution.SparkAcidSqlParser.parsePlan(SparkAcidSqlParser.scala:56)
Line 56 doesn't have function call to parsePlan. Please recheck your jars.

@amoghmargoor below is from master branch, SparkAcidSqlParser.scala:56 is part of parsePlan function. Am I missing anything?

@srinikvv
I think you are reading the stack trace wrong. If you look carefully at stack trace it is saying:
SparkSession.sql(SparkSession.scala:642) calls Line 56 of SparkAcidSqlParser i.e., SparkAcidSqlParser.parsePlan(SparkAcidSqlParser.scala:56). SparkAcidSqlParser.parsePlan(SparkAcidSqlParser.scala:56) will call AbstractSqlParser.parsePlan and that will call SparkSqlParser.parse and so on... So according to stack trace Line 56 of SparkAcidSqlParser.scala should have a function call to parsePlan but instead it has throw statement which means you are using old code still.

hey @srinikvv ... were you able to fix your issue ?

@amoghmargoor your suspicion is correct, spark-shell was caching the spark-acid-assembly.jar. I was able to test using pySpark and conf "spark.driver.userClassPathFirst=true". However, I see below error while trying to activate the extension:
>>> spark.sparkContext._jvm.com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension() \
...             .apply(spark._jsparkSession.extensions())
********HiveAcidAutoConvertExtension***********
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/spark-2.4.3-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o35.apply.
: java.lang.LinkageError: loader constraint violation: when resolving method "org.apache.spark.sql.SparkSessionExtensions.injectResolutionRule(Lscala/Function1;)V" the class loader (instance of org/apache/spark/util/ChildFirstURLClassLoader) of the current class, com/qubole/spark/hiveacid/HiveAcidAutoConvertExtension, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, org/apache/spark/sql/SparkSessionExtensions, have different Class objects for the type scala/Function1 used in the signature
        at com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension.apply(HiveAcidAutoConvert.scala:72)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:745)

oh ... something like this just works for me:
bin/pyspark --master local[1] --jars /Users/amoghm/src/apache/spark-acid/target/scala-2.11/spark-acid-assembly-0.5.0.jar --conf spark.sql.extensions=com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension

@amoghmargoor I was finally able to get this working after downloading Hadoop2.8.2 binaries and setting SPARK_DIST_CLASSPATH to reference these libraries.
However the MERGE syntax only works with ACID tables which are not bucketed and I see below exception when used against ACID bucketed tables:
        at com.qubole.spark.hiveacid.HiveAcidErrors$.unsupportedOperationTypeBucketedTable(HiveAcidErrors.scala:53)
        at com.qubole.spark.hiveacid.HiveAcidOperationDelegate.checkForSupport(AcidOperationDelegate.scala:471)```

As per the documentation on [Apache Hive Confluence](https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Limitations) all ACID tables must be bucketed. Hence MERGE statement on Spark-Acid without support for bucketed tables is hindering the use of this feature.
Do you have any plans in the near future to support MERGE on bucketed tables?

Hi @srinikvv ,
good to hear you could fix the issue. is this something we can add to FAQ or Troubleshooting guide that can help others ?
Btw regarding bucketed tables, Hive ACID does not require them anymore. This was the restriction with earlier implementation of Acid which was changed and now this restriction does not hold. We create non-bucketed Hive ACID table all the time internally in Qubole. If you are using Hive 3.1 and onwards you should be good. We did not add bucketed table support because bucketing hashes are not same across engine.

@amoghmargoor Yes this can be added to FAQ or Troubleshooting. I believe the issue is when using a spark binaries compiled with hadoop libraries < 2.8.2 (was using spark-2.4.3-bin-hadoop2.7). As a workaround we have downloaded hadoop 2.8.2 libraries and setting SPARK_DIST_CLASSPATH to refer the new hadoop libraries as below before running the spark-submit command:
export SPARK_DIST_CLASSPATH=$(<path_to_hadoop_2.8.2_bin>/hadoop classpath)
A better approach may be is to build a spark 2.4.3 with Hadoop 2.8.2 binaries. I am currently trying this and let you know the result.
Also I tried to perform merge on non bucketed ACID tables, was facing org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException for a specific table, below is the stack trace:
 20/07/06 07:58:26 INFO MergeImpl: MERGE Clause 1: UPDATE being executed 20/07/06 07:58:26 INFO TableWriter: Write Operation being performed to table booking_f: UPDATE 20/07/06 07:59:16 WARN TaskSetManager: Lost task 50.0 in stage 10.0 (TID 1753, hdop-stg-wrk2.vmware.com, executor 3): org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): /warehouse/tablespace/managed/hive/base.db/booking_f/delete_delta_0000011_0000011_0000/bucket_02359 path already being created, clientname=DFSClient_NONMAPREDUCE_1168652467_87 	at org.apache.hadoop.ipc.Client.call(Client.java:1475) 	at org.apache.hadoop.ipc.Client.call(Client.java:1412) 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) 	at com.sun.proxy.$Proxy15.create(Unknown Source) 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 	at java.lang.reflect.Method.invoke(Method.java:498) 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) 	at com.sun.proxy.$Proxy16.create(Unknown Source) 	at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648) 	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689) 	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624) 	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) 	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) 	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459) 	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) 	at com.qubole.shaded.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95) 	at com.qubole.shaded.orc.impl.WriterImpl.<init>(WriterImpl.java:177) 	at com.qubole.shaded.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:94) 	at com.qubole.shaded.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:334) 	at com.qubole.shaded.hadoop.hive.ql.io.orc.OrcRecordUpdater.addSplitUpdateEvent(OrcRecordUpdater.java:456) 	at com.qubole.shaded.hadoop.hive.ql.io.orc.OrcRecordUpdater.update(OrcRecordUpdater.java:498) 	at com.qubole.spark.hiveacid.writer.hive.HiveAcidFullAcidWriter.process(HiveAcidWriter.scala:275) 	at com.qubole.spark.hiveacid.writer.TableWriter$$anon$1$$anonfun$apply$1.apply(TableWriter.scala:146) 	at com.qubole.spark.hiveacid.writer.TableWriter$$anon$1$$anonfun$apply$1.apply(TableWriter.scala:146) 	at scala.collection.Iterator$class.foreach(Iterator.scala:891) 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) 	at com.qubole.spark.hiveacid.writer.TableWriter$$anon$1.apply(TableWriter.scala:146) 	at com.qubole.spark.hiveacid.writer.TableWriter$$anon$1.apply(TableWriter.scala:138) 	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101) 	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101) 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) 	at org.apache.spark.scheduler.Task.run(Task.scala:121) 	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 	at java.lang.Thread.run(Thread.java:745)
This looks similar to 41. However I realized that the merge statement is failing while trying to create a delete_delta folder. Can you assist me here?

@amoghmargoor is there a plan to modify the plugin to work on Spark 3?
We want to explore some feature of Spark3 and it also comes pre-compiled with Hadoop 3.2 which can help avoid the earlier issue out of the box.
FYI: I already tried initializing spark-acid plugin with Spark 3 but it fails with below error:
Py4JJavaError: An error occurred while calling o28.sessionState. : java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class         at com.qubole.spark.datasources.hiveacid.sql.execution.SparkAcidSqlParser.<init>(SparkAcidSqlParser.scala:20)         at com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension$$anonfun$apply$3.apply(HiveAcidAutoConvert.scala:74)         at com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension$$anonfun$apply$3.apply(HiveAcidAutoConvert.scala:73)         at org.apache.spark.sql.SparkSessionExtensions.$anonfun$buildParser$1(SparkSessionExtensions.scala:205)         at scala.collection.IndexedSeqOptimized.foldLeft(IndexedSeqOptimized.scala:60)         at scala.collection.IndexedSeqOptimized.foldLeft$(IndexedSeqOptimized.scala:68)         at scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:49)         at org.apache.spark.sql.SparkSessionExtensions.buildParser(SparkSessionExtensions.scala:204)         at org.apache.spark.sql.internal.BaseSessionStateBuilder.sqlParser$lzycompute(BaseSessionStateBuilder.scala:129)         at org.apache.spark.sql.internal.BaseSessionStateBuilder.sqlParser(BaseSessionStateBuilder.scala:128)         at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:329)         at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1107)         at org.apache.spark.sql.SparkSession.$anonfun$sessionState$2(SparkSession.scala:157)         at scala.Option.getOrElse(Option.scala:189)         at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:155)         at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:152)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)         at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)         at py4j.Gateway.invoke(Gateway.java:282)         at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)         at py4j.commands.CallCommand.execute(CallCommand.java:79)         at py4j.GatewayConnection.run(GatewayConnection.java:238)         at java.lang.Thread.run(Thread.java:745)

I also wanted to report another error I observed while trying to perform merge using a spark table (RDD based table created using df.createOrReplaceTempView) as source.
An error occurred while calling o82.sql. : java.lang.UnsupportedOperationException: Update Dataframe doesn't have expected schema.  Provided: StructField(rowId,StructType(StructField(writeId,LongType,true), StructField(bucketId,IntegerType,true), StructField(rowId,LongType,true)),true),StructField(booking_id,LongType,true),StructField(edw_create_date,TimestampType,true),StructField(edw_update_date,TimestampType,false)  Expected: StructField(rowId,StructType(StructField(writeId,LongType,true), StructField(bucketId,IntegerType,true), StructField(rowId,LongType,true)),true),StructField(booking_id,LongType,true),StructField(edw_create_date,TimestampType,true),StructField(edw_update_date,TimestampType,true) 	at com.qubole.spark.hiveacid.HiveAcidOperationDelegate.mergeUpdate(AcidOperationDelegate.scala:403) 	at com.qubole.spark.hiveacid.merge.MergeImpl$$anonfun$runMergeOperations$2.apply(MergeImpl.scala:218) 	at com.qubole.spark.hiveacid.merge.MergeImpl$$anonfun$runMergeOperations$2.apply(MergeImpl.scala:210) 	at scala.collection.Iterator$class.foreach(Iterator.scala:891) 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) 	at scala.collection.IterableViewLike$Transformed$class.foreach(IterableViewLike.scala:44) 	at scala.collection.SeqViewLike$AbstractTransformed.foreach(SeqViewLike.scala:37) 	at com.qubole.spark.hiveacid.merge.MergeImpl.runMergeOperations(MergeImpl.scala:210) 	at com.qubole.spark.hiveacid.merge.MergeImpl.run(MergeImpl.scala:196) 	at com.qubole.spark.hiveacid.HiveAcidOperationDelegate.merge(AcidOperationDelegate.scala:444) 	at com.qubole.spark.hiveacid.HiveAcidTable$$anonfun$merge$1.apply(HiveAcidTable.scala:199) 	at com.qubole.spark.hiveacid.HiveAcidTable$$anonfun$merge$1.apply(HiveAcidTable.scala:199) 	at com.qubole.spark.hiveacid.HiveTxnWrapper.inTxnRetry$1(HiveAcidTable.scala:330) 	at com.qubole.spark.hiveacid.HiveTxnWrapper.inTxn(HiveAcidTable.scala:355) 	at com.qubole.spark.hiveacid.HiveAcidTable.merge(HiveAcidTable.scala:198) 	at com.qubole.spark.hiveacid.datasource.HiveAcidRelation.merge(HiveAcidRelation.scala:98) 	at com.qubole.spark.datasources.hiveacid.sql.catalyst.plans.command.MergeCommand.run(MergeCommand.scala:51) 	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) 	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) 	at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79) 	at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194) 	at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194) 	at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364) 	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78) 	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125) 	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73) 	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363) 	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194) 	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79) 	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642) 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 	at java.lang.reflect.Method.invoke(Method.java:498) 	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) 	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) 	at py4j.Gateway.invoke(Gateway.java:282) 	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) 	at py4j.commands.CallCommand.execute(CallCommand.java:79) 	at py4j.GatewayConnection.run(GatewayConnection.java:238) 	at java.lang.Thread.run(Thread.java:745)
The error suggests that the schema of the source and the target does not match. However the only discrepancy is that the column edw_update_date was nullable in Target but this column was passed as current_timestamp() in source, which makes it a not null column in source schema. This however should not cause any error while performing the merge.
@amoghmargoor please let me know if you need me to open another issue for this

@srinikvv W.r.t org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException, is this being thrown on task retries ? similar to issue #43 ?
reg, Spark 3 we are yet to start work on it. My guess is we might start looking at it July end.
reg the type issue, yes data transfer from non-nullable to nullable should be allowed. we will take a look at it. You can get unblocked by explicitly assigning schema to your source DF that matches target schema.

hey @srinikvv , how are things ? were you able to get MERGE working ?

@srinikvv W.r.t org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException, is this being thrown on task retries ? similar to issue #43 ? That issue has been fixed now.
reg, Spark 3 we are yet to start work on it. My guess is we might start looking at it July end.
reg the type issue, yes data transfer from non-nullable to nullable should be allowed. we will take a look at it. You can get unblocked by explicitly assigning schema to your source DF that matches target schema.

@amoghmargoor I built a latest jar with code from master branch on 14-Jul and retested the merge statement failing with org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException. I still get this error for only a specific table:

Perform MERGE on an empty table with full dateset --> completes successfully (approx 90M records)
Wait until compaction is successful on this table
Perform MERGE using a only changed data (13K records). Fails during the Update step with below error message
I believe that multiple Tasks are trying to write to the same file, not sure how to avoid this error.
Note: The same Merge statement is running fine from Hive.

20/07/15 14:55:08 INFO MergeImpl: MERGE requires right outer join between Target and Source.
20/07/15 14:55:08 INFO MergeImpl: MERGE Clause 1: UPDATE being executed
20/07/15 14:55:08 INFO TableWriter: Write Operation being performed to table service_contract_f: UPDATE
20/07/15 14:55:12 WARN TaskSetManager: Lost task 132.0 in stage 45.0 (TID 2473, hdop-stg-wrk12.vmware.com, executor 9): org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): /warehouse/tablespace/managed/hive/base.db/service_contract_f/delete_delta_0000027_0000027_0000/bucket_00004 path already being created, clientname=DFSClient_NONMAPREDUCE_-1263939954_80
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493)
        at org.apache.hadoop.ipc.Client.call(Client.java:1439)
        at org.apache.hadoop.ipc.Client.call(Client.java:1349)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
        at com.sun.proxy.$Proxy15.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:297)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy16.create(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:269)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1274)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1216)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:477)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:474)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:474)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:415)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1067)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1048)
        at com.qubole.shaded.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:95)
        at com.qubole.shaded.orc.impl.WriterImpl.<init>(WriterImpl.java:177)
        at com.qubole.shaded.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:94)
        at com.qubole.shaded.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:334)
        at com.qubole.shaded.hadoop.hive.ql.io.orc.OrcRecordUpdater.addSplitUpdateEvent(OrcRecordUpdater.java:456)
        at com.qubole.shaded.hadoop.hive.ql.io.orc.OrcRecordUpdater.update(OrcRecordUpdater.java:498)
        at com.qubole.spark.hiveacid.writer.hive.HiveAcidFullAcidWriter.process(HiveAcidWriter.scala:292)
        at com.qubole.spark.hiveacid.writer.TableWriter$$anon$1$$anonfun$apply$1.apply(TableWriter.scala:146)
        at com.qubole.spark.hiveacid.writer.TableWriter$$anon$1$$anonfun$apply$1.apply(TableWriter.scala:146)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.foreach(WholeStageCodegenExec.scala:634)
        at com.qubole.spark.hiveacid.writer.TableWriter$$anon$1.apply(TableWriter.scala:146)
        at com.qubole.spark.hiveacid.writer.TableWriter$$anon$1.apply(TableWriter.scala:138)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:121)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Please check this and let me know if I can do anything to fix this issue.

@amoghmargoor pls let me know if we can do a Zoom session to show you this issue

@srinikvv that would be great. I and @sourabh912  are in PST timezone and would like to join the call. Send some time slots that would work for you guys. also feel free to join the group<EMAIL_ADDRESS>

@amoghmargoor we can meet at 16July IST 9:00PM to 10:00PM, if that works for you guys.

hey @srinikvv I missed the message above. This timing should work for me on Friday (but I guess it would be Friday night for you guys). Or else on Monday too I would be available for the call. Let me know if that would be fine.

@amoghmargoor lets meet Today. please use below zoom link:
Topic: Veera Venkata Rao's Zoom Meeting
Time: Jul 17, 2020 09:00 PM Mumbai, Kolkata, New Delhi
Join Zoom Meeting
https://VMware.zoom.us/j/98812494211?pwd=cjhqRktESTJ0L0l3elJxaVJ1YVMxUT09
Meeting ID: 988 1249 4211
Password: 494932

sure ... see you at 8:30 am PST.
Sent from my iPhone

On 17-Jul-2020, at 5:18 AM, srinikvv<EMAIL_ADDRESS>wrote:

@amoghmargoor lets meet Today. please use below zoom link:
Topic: Veera Venkata Rao's Zoom Meeting
Time: Jul 17, 2020 09:00 PM Mumbai, Kolkata, New Delhi
Join Zoom Meeting
https://VMware.zoom.us/j/98812494211?pwd=cjhqRktESTJ0L0l3elJxaVJ1YVMxUT09
Meeting ID: 988 1249 4211
Password: 494932
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

@amoghmargoor Appreciate you guys taking time to understand/debug the issue.
As discussed, please let us know the steps to perform update using an API. Also if you the Debug Jar or Branch created with appropriate logger statements to help fix the issue.

@srinikvv I think existing update API may not be able to support your use case. Can you try running by compiling the jars from here: https://github.com/amoghmargoor/spark-acid/pull/new/issue-70 ? I have added debug logs. Run it with just 2 executors and provide me logs for both the driver and executor after the failure. I may followup with few more such iterations. Thanks.

@amoghmargoor shared the logs via email. Reg Speculative execution, we are executing jobs with default value for spark.speculation (which is false for version Spark2.4.3)

@srinikvv We have figured out an issue why this could be happening and I have added the fix here: https://github.com/amoghmargoor/spark-acid/pull/new/issue-70. Can you recreate jar from the branch above and check if your issue has been fixed by it ?

@srinikvv : Thanks again for reporting the issue. As part of this we identified few issues (issue #83 and issue #93) and fixed them.
I am closing this one. Please feel free to reopen it or open a new issue if you face any problem.


post or PUT http request with list of objects as parameters

  var teacher=[teachers]()   // imagine this array is feeded

guard let url=URL(string: "google.com") else {return}
    var request=URLRequest(url: url)
    request.httpMethod="PUT"

    request.addValue("Application/Json", forHTTPHeaderField: "Content-Type")

guard let httpbody=try? JSONSerialization.data(withJSONObject: teacher.toJSON(), options: []) else{return true}
    request.httpBody = httpbody

    URLSession.shared.dataTask(with: request) { (data, response, Error) in

        if let data=data {
let convertedString=String(data: data, encoding: String.Encoding.utf8)
            print(convertedString)

}
the exception is  :
"status":400,"error":"Bad Request","exception":"org.springframework.http.converter.HttpMessageNotReadableException","message":"JSON parse error: Can not deserialize

you can print json see see.


Adding support for Shapeshift

I think that integrate Shapeshift in the wallet to both deposit and withdraw will make a huge start of the beta, by reaching the entire cryptocurrencies community instantly.

Agree, Coinbase buy widget would be good too

Moved to CH


Luigi doesn't check default arguments when deciding whether a job is already running

I have a job with a date CLI argument that defaults to yesterday. It's cronned to run daily.
However, if there's a problem with the job and it doesn't terminate for one day, then the next day when the command is run, luigi reports that the job is already running, when in fact it isn't because the date parameter will now be different.
E.g.:
The job is invoked on 2015-03-30 with my-job --another-arg=blah. The date parameter is not explicitly passed so defaults to the day before, 2015-03-29. This job fails and retry logic keeps it around.
The next night, 2015-03-31, cron executes my-job --another-arg=blah again, and this time the date parameter defaults to the value 2015-03-30. Luigi should run this job but it doesn't. Instead it reports that the job is already running and exits.
Obviously the fix for now is to explicitly pass the date parameter, but really luigi should calculate the values of all default values before deciding whether a job is already running or not.

I'm closing this issue.

[x] It has been inactive for +4 months.
[ ] It's not about luigi core, so not as many users are affected about this.
[ ] The change seems quite big, it's unlikely to be sporadically picked up.
[ ] The owner haven't responded or disappeared.
[x] I don't understand what this issue is about.
[ ] There exists a reasonable workaround for this.
[x] We need to check if this hasn't been fixed by now (for old issues).
[ ] This is kind of by design and not a bug.
[ ] Resolving this would probably add a lot of complexity.

Every open issue adds some clutter, and we try to make the issues fewer and make it easier for new collaborators to find. Currently we try to close any issue that meets the first checkbox + one other.
Feel free to reopen this issue if the problem still persists.


feat(controller): more git pkg enhancements

This adds a couple more enhancements to our git package:

On clone, persists preference for ignoring cert errors as config so that preference does not need to be re-specified when loading an existing repo from the FS.

Add option when pushing to push to a remote branch with a different name than the current local branch.

There are more prerequisite capabilities for the git-based promotion directives.

On clone, persists preference for ignoring cert errors as config so that preference does not need to be re-specified when loading an existing repo from the FS.

Is there a hypothetical scenario where this may change between attempts?

Is there a hypothetical scenario where this may change between attempts?

I thought about that as well, but it seemed that this would require both the validity of a cert to change between directives/attempts and for the desire to ignore/not ignore invalid certs to also change as well. The former seems like an edge case, but possible. The latter seems impossible, however, since I assume the directives from which one Promotion is composed cannot be mutated once that promotion is in-flight.
So I think that when you clone, you're either ignoring or not ignoring cert errors for the life of that clone.
wdyt?

Think the latter observation solves all of my initial concerns.


The deleted Light Group is still displayed on Gateway3 with the status of Unavailable

After removing Light Group in mi home, there still exists a device Xiaomi Mesh Group in gateway3 plugin, it shows unavailable, restarting Home Assistant several times is invalid, sometimes adding Light Groups composed of different brands of lights do not show up in gateway3, in fact, they work fine in mi home! Works fine
Gateway used: Xiaomi Multimode Gateway 2
Firmware version: 1.0.3_0028

Try to delete them manually. Component didn't delete devices automatically.

Manual deletion works, but a new problem, I have four spotlights of the same brand and a strip light controller to form a Light Group, they work fine in the meter home, but won't show up in homeassistant unless I form the four spotlights into a Light Group without adding the strip light controller!


Limit query and return a Results

Goals

I need to limit the size of a query, but since Realm loads the data on demand, the only way I can see to do this is to create an Array of Object, which is not good in my case because I need to deal with a Results<T> and ThreadSafeReference. So what's the best way to limit a query and return a Results<T>? I can't see any solution for this with the current implementation.
I am trying to find a solution using double query, but since my object doesn't have a primaryKey, this is being difficult.
I tried to do this but the filter is crashing:
let all: Results<Foo> = realm.objects(Foo.self)
let limit: [Foo] = all.suffix(100) //Get the last 100 items
let result: Results<Foo> = realm.objects(Foo.self).filter("SELF IN %@", limit) //Crash

This is impossible since Realm doesn't support it: https://stackoverflow.com/a/32568366/2430555
I need another solution, any idea?
Edit:
Is there any way, illegally, to init Results with populated data?
Version of Realm and Tooling

Realm framework version: 3.15.0
Xcode version: 10.2.1
iOS/OSX version: Any
Dependency manager + version: Cocoapods

@ppamorim Why do you think you need to limit the query? How many results do you get without limits?

@bmunkholm I need to get at maximum 100 items and save these on a ThreadSafeReference because they are deleted after the completion of an IO in another thread.


How to achieve both 'block' and 'inline' Markdown highlighting?

Hi Marcin, thanks for all the incredible TextKit-related work you've done — and shared. Your projects have been an invaluable resource.
Issue
When assigning language: .markdown as the NeonPlugin language, only block grammar such as fenced code blocks, quotes, lists, headings etc are styled.
As I understand it, TreeSitter can/does use a two-pass parse (ha) for markdown, for block and inline grammar? Is this correct? If so, will I need to implement my own code to achieve this second parsing operation — or indeed, is this even possible? Based on my very tentative TextKit intuition, this sounds expensive. But perhaps (hopefully!) I'm just missing a trick.
Reproduction
I have created an STTextView as per the basic defaults, as outlined in the readme's for this and the main project.
STTextViewUI.TextView(
    text: $text,
    selection: $selection,
    options: [.wrapLines, .highlightSelectedLine],
    plugins: [NeonPlugin(theme: .default, language: .markdown)]
)

My hope was to see such strings as "inline code", "italic text" etc styled in the typical fashion, but only the block elements mentioned above are receiving styling.
Thanks in advance for any help you can provide.

Keen on this too, but I have no idea how to do it. Would you have any pointers Marcin?

Good point. I'll investigate with Neon and/or tree-sitter and report back in this thread if I find something to do in STTextView-Plugin-Neon too 👍

Based on a few hours of investigation and reading code, my understanding is as follows:

The implementation/wiring will reside in the STPluginNeon plugin. Neon only offers the platform to integrate "token providers" (among which TreeSitter languages are one option), and TreeSitter only offers the standardised grammars for each language.
The TreeSitter Markdown grammar explicitly mentions doing a 2 pass for markdown block and markdown inline (see penultimate paragraph of README here.)
STPluginNeon already includes the grammars for markdown (block) and markdown inline.
I noticed the TreeSitter markdown grammar mentions markdown_inline as a "language" that can be injected in markdown (like HTML and YAML can be injected in Markdown). However it seems the current STPluginNeon implementation doesn't leverage the injection queries?

Could the injection queries be the mechanism to achieve the second pass? By looking at the git history of the tree-sitter-markdown's repo, I have the impression the comment at the bottom of the README may be outdated, and the code comments at the top of their sample implementation (in bindings/rust/lib.rs) seems to also be out of date with the code.
I also had a look at Simon B. Støvring's Runestone iOS app. It successfully does Markdown block and inline highlighting. I had a look at his Runestone and TreeSitterLanguages repos but couldn't find evidence of a "manual" two pass. I then followed his documentation to integrate Runestone (the text view) into a sample iOS app, and added syntax highlighting for markdown. The result is that it doesn't highlight inline markdown automatically...
Conclusion
I have the feeling that injection alone is not sufficient, and achieving a good block & inline highlighting is probably down to some sort of "special case", or at least some advanced understanding of how TreeSitter works. Knowledge I don't have.
@krzyzanowskim, does this word salad make sense to you 😅?
I'd be happy to help write the implementation but the domain knowledge is quite far above my head, I'd need to (initial) guidance. Alternatively, Simon B. Støvring is probably doing this in the closed source part of the iOS app, we could ask him for pointers?
I hope this helps 🙂

Sweet, thank you @mattmassicotte for chiming in!
I played a bit with the sample project you linked and indeed managed to have a sample MD string to work 👍
(I couldn't highlight properly some "emphasis" and "strong emphasis" but I don't know if it's because of my quick & hacky hardcoded token styling, or from the grammar itself? Anyway, that's secondary to the main effort in this thread).
I barely understand the code in the plugin, let alone in Neon or TreeSitter... So I don't feel qualified to propose a PR just yet 😅
@krzyzanowskim let me know what's your recommendation/approach in terms of architecture in the plugin?

I'd be happy to look more closely with you though!

Sure! How do you want to go about it?

@jeanetienne Here I've opened an issue: https://github.com/ChimeHQ/Neon/issues/51


[master] Upgrade to latest dependencies

Cron. -knative-prow-robot
/cc knative-sandbox/source-wg-leads
/assign knative-sandbox/source-wg-leads
Produced via:
./hack/update-deps.sh --upgrade --release v0.21
./hack/update-codegen.sh

Details:
bumping knative.dev/serving 96f41b5...cc0209f:
  > cc0209f Remove outdated todo's (# 10804)
  > 493918e Update net-contour nightly (# 10801)
  > 87bf1a9 Update net-kourier nightly (# 10800)
  > 7f9dd35 Update net-certmanager nightly (# 10799)
  > c24dc14 Validate domain mapping is not a cluster local name (# 10798)
  > 7aa62c5 Reflow the code to follow a better builder pattern. (# 10796)
  > 317d055 Remove the last check for percentage to be nil (which it can't anymore). (# 10797)
  > c6a28b7 Update common github actions (# 10794)
bumping knative.dev/eventing 84b99db...c44a698:
  > c44a698 upgrade to latest dependencies (# 4896)
  > e063a8d update default-pingsource configmap checksum (# 4895)
  > fb8e3b4 Fixup ping source dataMaxSize configmap (# 4894)
  > 78ee789 Adding checkRetry from kafkaChannel (# 4890)
  > 30d9fa3 Format markdown (# 4891)
  > dbdb339 [doc] Debugging Knative controllers and friends locally with Intellij Idea (# 4886)
  > ef5f61d Update common github actions (# 4888)

/approve
/lgtm


validation errors mapped to observations

If and when a validation fails for some reason (improper formatting, typos, etc) We don't want one validation to halt the execution of an assessment - but we do want to capture some meaningful data for debugging.
In addition to logging, Lula should report back errors in a specific validation mapped back to an observation that is also marked as not-satisfied. For something like "No resources found etc" this can be valuable end-user information to guide what occurred.
More so in the event that Lula output is tied to a GRC system.

Discussed - this should be bumped to prevent full-execution exiting


No output from Devika

Devika doesn't respond. Any ideas?

try creating new project and try again, Wait for a while after first query
I see only the first time , agent execution is called in the api

try creating new project and try again, Wait for a while after first query I see only the first time , agent execution is called in the api

Hmm, it says the api_key is missing - it's set though.

Yep thats right , see issue: https://github.com/stitionai/devika/issues/29

Fixed here: #59

fetch the latest changes. already fixed