Really bad first-time user experience

Really bad first-time user experience
Hi there. Yesterday I gave JupyterLab Desktop a try and had a bad and off-putting experience. I wanted to use that experience to  provide feedback to the developers to help improving the app.
I know this is not a bug report in itself, but I decided on sharing this story here because I think that some of my points here maybe could be converted into bug reports or feature requests. But because it is so much, I wasn't sure whether it makes sense to just create a bunch of issues all at once. It's probably better if this is looked at by a developer or experienced user who can decide what's helpful to record as issues, what's already been reported and what should just be ignored.
Context: This experience is all from the first ~3 hours of using JupyterLab and I haven't put more energy into it after that. I've looked into similar projects over the past 10-or-so years, e.g. the IPython QT frontend, nteract, Jupyter notebook support in PyCharm. I never invested more than 30 minutes into one of these tools because it didn't work out for me. I'm an active daily user of Mathematica 12 and IntelliJ IDEA 2022.
To improve readability, while reporting my experience, whenever something went wrong, I'll insert quote blocks to describe what I would have expected to happen or how I think that the app's behavior could be improved.

I downloaded the macOS app from JupyterLab Desktop's GitHub page after getting there from the blog post JupyterLab Desktop App now available!. There I found the link labelled macOS Installer.

This made me question whether I want to continue. I had a lot of bad experiences with macOS apps requiring an installer. Expecting a bad experience here too, I questioned whether I have the energy to continue with that. The link doesn't actually lead to an installer but instead to a DMG containing the app, so it could be labelled e.g. simply macOS.

I launched the app for the first time and was shown this dialog:

I wanted to re-use my existing environment because I'm low on disk space. So I selected Select Python path to open a file browser. I had the path in my clipboard already, so and I hit command-shift-G and command-V to navigate there. This popped up the following error message:

It's a minor thing but having this work as expected would have improved the user experience for me.
Before I opened the file browser, I wondered why I couldn't paste the path from my clipboard directly into the field with an orange border. It looks like a text input field by the way of having an I-beam cursor when moving the mouse over it, but activating it and hitting command-V did nothing. If it's not a text input field then maybe it should be styled differently.
Personally I would completely redo this dialog. It has 7 interactive controls to choose between 3 actions (quit, install bundled environment, use existing environment). So it could instead just have three buttons for those actions (e.g. "Quit", "Use bundled environment", "Choose existing environment"). The latter of those buttons would open the file browser and continue with re-launching the application when a path is selected.

So I navigated to the path manually and then selected Apply and restart, but I hit #473 in the process. Investigating this cost me about 45 minutes (clearing and re-installing dependencies a few times) until I gave up and set it to Install Python environment using the bundled installer and continued.

When something fails while setting up the environment, the app shouldn't simply continue with restarting and instead display an error, even if it's just a stack trace.

JupyterLab finished launching, and I was greeted with this launcher (missing the Notebook options):

I don't know whether this is an expected state for the app to be in and whether it makes sense to handle it, but it would have helped me a lot in this case if just a simple message was displayed like "Hey! You don't have any kernels installed, that's why there isn't an option to create a notebook, follow this link for instructions on how to add a kernel."

I didn't know that there was anything wrong yet, so I created a new notebook via File > New > Notebook and got this dialog:

The drop-down menu was inactive, so I had no other option than to leave it at No Kernel. I could enter expressions into the created notebook but not evaluate it. So I thought I have to figure out how to add a kernel. I couldn't find an option for that in any of the menus or anywhere else in the UI. I searched the web with queries like "jupyterlab desktop add kernel" but couldn't find anything that helped me. Then I tried adding a kernel from the command line using /Users/michi/Library/jupyterlab-desktop/jlab_server/bin/jupyter-kernel install which allowed me to figure out the problem that prevented the Python kernel from being added automatically. This took another hour to figure out.

Again, a simple message for that case would have saved me a lot of time and energy.

I solved the underlying problem, closed the app's window and double-clicked the notebook. The dialog for selecting a kernel was shown again, this time with the Python 3 kernel pre-selected (yay for automatically recognizing that it is possible to add the kernel now 👍). I chose Select and tried to evaluate an expression, which didn't work. Instead, the cell stayed in this state:

The status bar said Connecting and stayed in that state for a long time:

After some time it changed to Disconnected, so I selected Kernel > Reconnect to Kernel from the main menu, which changed the state back to Connecting, but without success. So I quit the application and re-launched it. After that, it was able to connect to the kernel.

While writing up my experience here, I had to figure out that the issue was caused by me not quitting the application and instead just re-opening the notebook from Finder. It's obvious to me that re-launching the application could be required after fixing an issue that prevented the kernel from being added but for some reason I didn't do it at that time and my experience was really bad because of that.
I think this case could be handled more gracefully. If an error message was displayed instead of stalling in the Connecting state, I would have been more inclined to re-launch the application right away (which would have been a better experience because I wouldn't have had the feeling that I was wasting my time).
Better of course would be if this problem wouldn't occur at all (at it just worked) or if the app would notice that a re-launch was necessary and tell me so.
(I later noticed while writing this down that this condition happens from time to time even after the application has been set up already, so this bug isn't isolated to that case)

I was able to evaluate expressions now. Now I wanted to move my notebook to a different location.

IMHO the user's home directory is a bad choice as a default location for new notebooks.

I saved the notebook, moved it in Finder to the new location deep down in my Documents folder and renamed it there. Then I switched back to the app and noticed that (a) the integrated file browser in the left panel didn't pick up that the notebook is no longer in my home directory and (b) that it still displayed the notebook's old name in the tab and title bar's.

It should just do (a) and (b).

I couldn't find a way to refresh the integrated file browser, so I hit command-R, which re-loaded the whole window. This lost me some changes I had in another notebook without warning.

To reproduce, click into the file browser and hit command-R. The window will reload without the usual Do you want to leave? message. This isn't ok.

Now I wanted to navigate the integrated file browser to the new location of my notebook. I did the usual thing on macOS and dragged the folder from Finder (the little proxy icon in the folder window's title bar) onto the integrated file browser. That resulted in this error message:

I didn't know about the Show in File Browser option in the notebook's tab's context menu at that time. My expectation is that dragging a file or folder onto an application's file browser navigates the browser to that file. Handling it like I wanted to make a copy of the folder doesn't make sense to me in a desktop application. Instead, there should be a Reveal in Finder context menu to allow people to use the Finder to copy and move stuff around.
I know that some IDEs handle it this way, that you can drag stuff onto the file browser and have it copied or moved there, but there's a reason that I had a different expectation in this case: I didn't really open a "project" within the app, it simply launched and showed me the contents of my home directory, the same way that e.g. a GUI Git client would show a file browser to select a Git repository on first launch (where I would be able to drag something from Finder to navigate).

So I had the file open twice now. I switched to the old tab and hit command-W. The whole window simply disappeared.

I would have expected command-W to close a single tab instead.

And now the app was in a weird state. Without an open window, there also wasn't a useful menu bar. Opening the notebook from Finder again worked though.

The menu bar should just be moved out of the window into the actual menu bar. I've already seen an issue here requesting that.
This is also about the time I noticed how long JupyterLab takes to launch or open a new window. On my Mac, it takes about 18 seconds when launching the application via double-clicking a notebook from Finder until notebook's content is displayed. As a comparison, Mathematica 12 takes about 4 seconds and IntelliJ IDEA 2022 about 8 seconds.

I finally started writing some code. While doing that, I accidentally hit command-,, which opened the app's settings in a new tab. To close it again, I hit command-W again, which prompted this dialog to show up:

I read Do you want to leave? and though "no", so I hit escape. This closed the window again. I re-opener my notebook via Finder and noticed that some of my changes had been lost.

That this dialog exists makes no sense to me. If the app thinks that I want to close all my open notebooks, it should simply ask for whether I want to save them. A dialog for that does exist, but isn't used here!
Hitting escape shouldn't lead to a destructive action.
 But it happened countless times by accident while and after complaining about it here 😭.
Also, I have Autosave Documents activated in the Settings menu. Why didn't it work here?! (I notice now that talking about an app losing some of my work because of bad UI design and/or bugs raises emotions in me, I hope it won't detract from the constructive messages I want to convey here. Bear with me).
Also, I would prefer it if notebooks would simply be saved whenever the window looses focus or when the tab/window/application is closed. I don't see a compelling reason to differ in behavior here from most other IDEs and macOS apps and also from practically all web applications (when considering that the UI was originally designed to be used as a web app).
This is also the point where I noticed that the app wouldn't remember the size and location of the main window between launches, which is a bit annoying.

Then I got annoyed about the editor not wrapping the code to the window's width, so I went looking for an option for that. The settings' built-in search lead me to believe that there is no such option:

I know that searching a collection of dialogs for a search string isn't trivial to implement, but having a search field but not working as expected is worse than not having the search field at all IMHO, because it may lead to wrong conclusions about the existence of a setting.

So I enabled word wrapping and got this:

Word wrapping without indenting wrapped lines is worse than no wrapping at all, so IMHO this option is practically useless for the code editor.
This is a feature that most other IDEs have. I'm accustomed to it thus would expect it to be supported in a new IDE I'm evaluating.

I wanted to enable word wrapping to make better use of my laptop's small screen. Without that, I thought I can also just decrease the font size a bit, but oh boy is that part of the settings dialog broken! Entering a single digit in the Font Size field leads to the settings being saved and the field losing focus. The Font Family field has the same problem. Try entering a long font name with that:

😐

This is where I gave up. It's clear to me that using the app is a hassle to me instead of a relief, sadly. I know that much of the behavior that I find odd stems from the UI being designed with a web-app in mind, but these odd behaviors are the reason I never found the JupyterLab web app appealing.
I'm sorry, I know it's a lot 😕. It took the better part of the day to write all this down. I did it because I hope it'll help this project move forward and improve and have a higher chance of being accepted by new users.
I'll gladly try to respond to any questions or elaborate on points, but it sometimes takes multiple weeks for me to find the right time concentrate on it again.
I also don't mind if my points here are being dismissed. I just tried to give a transparent insight into my initial experience with the app. None of this is political, religious or emotional to me (or at least I'm not expecting you to fix that in that case).
Happy to read your feedback!

Thank you for your valuable feedback. I am sorry that you had such an experience with this distribution of JupyterLab. I agree with your points.
It does appear that many of the errors (environment issue, kernel issue, connection issue) arise from a general problem during the installation of JupyterLab Desktop specifically on MacOS; it is absolutely not how it should work. I do not have sufficient experience with MacOS, but maybe @mbektas could help.
Regarding the problems with selection dialog:

I made a PR to make the path editable (https://github.com/jupyterlab/jupyterlab-desktop/pull/478).
I agree that this dialog could be reworked. I personally would see it as a list of auto-detected environments grouped by source. I opened https://github.com/jupyterlab/jupyterlab-desktop/issues/479 to described remedy that I would prefer - what do you think?

The remaining issues need to be fixed in JupyterLab core:

broken search for nested properties in new settings UI: https://github.com/jupyterlab/jupyterlab/issues/12800
word wrapping: https://github.com/jupyterlab/jupyterlab/issues/6881. I've now added a comment to that issue.
focus lost when typing in settings UI: this was reported in https://github.com/jupyterlab/jupyterlab/issues/12160 and fixed in https://github.com/jupyterlab/jupyterlab/pull/12470 which was backported to 3.4 branch (you were likely using 3.3 due to a backlog in releasing JupyterLab Desktop).

I really appreciate the time you dedicated to sharing your detailed and thoughtful feedback. Would you mind if I bring it up on the upcoming Jupyter community meetings: JupyterLab call on Wednesday and wider governance call on Friday?

@Feuermurmel thanks for the detailed feedback, the time you dedicated to testing and sharing with us. I have been aware of some of the issues you reported (Command + R refresh, Command + W and state when there are no windows open but app still running) but didn't get chance to fix yet. Before each release, I test the app at least on macOS, Windows and Ubuntu but I never ran into some of the major issues you mentioned such as no kernel problem. We will be addressing the issues you
mentioned in the upcoming versions.

I've lost about 2 hours on a similar path and am currently stuck at the kernel disconnecting. Knowing there are potentially more issues ahead of that, I'm calling it quits and uninstalling. I really like the idea of a dedicated notebooks app, but I've just been completely defeated by this.

closing this issue. there has been many enhancements to the project recently that greatly improved the user experience. if there are ongoing problems please create separate issues for each.


Improve Arbitrary[HttpVersion]

Our arbitraries aren't as good as they ought to be.  I'll be gradually reforming them as we pass through the model.

This DecodeSpec is getting to be a bother.


Code highlight has unnecessary vertical scroll bar (with fix)

A fix would be: In _highlights.scss, change "overflow: scroll;" to "overflow: auto;".

Before

After

We also faced such problem and ended up with clena solution. It may be helpful for others so, we write it down: Code Highlight On Jekyll-GitHub Page


Mobx6

See:  https://github.com/xh/toolbox/pull/445
Note that this branch is off of the existing react17 branch/PR, and incorporates those changes.  I think we should probably upgrade these together for testing purposes.  I would be fine to just merge this and abandon the react17 branch.
Some notes, beyond what is in the changelog:

I only did part of the change to inheritance in our core model/service classes.  I have a plan to bundle up ReactiveSupport/XhIdSupport/ManagedSupport/LoadSupport in to a new base class (e.g. "HoistBase" (?)).  I want to wait on that until we have gotten rid of @HoistComponent and class based components, which still require the mixin approach.
(I can explain why).

There is a bit of risk in this upgrade that applications might either forget to call makeObservable() in a model constructor when needed, or forget to modify a destroy method().  The latter is the most serious concern, and can cause parts of your app to simple become unreactive without any other warning!  I don't have a great way to mitigate either of these concerns.  One good trick in code review to make sure that whenever we import from '@xh/hoist/mobx' (for the annotations) we also always import makeObservable (which we will also need to call).

Hoist P/R Checklist
Pull request authors: Review and check off the below. Items that do not apply can also be
checked off to indicate they have been considered. If unclear if a step is relevant, please leave
unchecked and note in comments.

[x] Caught up with develop branch as of last change.
[x] Added CHANGELOG entry, or determined not required.
[x] Reviewed for breaking changes, added breaking-change label + CHANGELOG if so.
[x] Updated doc comments / prop-types, or determined not required.
[x] Reviewed and tested on Mobile, or determined not required.
[x] Created Toolbox branch / PR, or determined not required.

If your change is still a WIP, please use the "Create draft pull request" option in the split
button below to indicate it is not ready yet for a final review.

Pull request reviewers: when merging this P/R, please consider using a squash commit to
collapse multiple intermediate commits into a single commit representing the overall feature
change. This helps keep the commit log clean and easy to scan across releases. PRs containing a
single commit should be rebased when possible.

We can't just put a call to makeObservable(this) in the HoistModel ctor? Does that not work because of class initialization order or something? If it doesn't work, could we put that call in a setTimeout or is there some reason it absolutely needs to be in the constructor?
Just trying to brainstorm some ways to avoid requiring all models to add a ctor and remember to make that call ... especially since it seems like it will not be necessary forever if they would finally approve a decorators proposal for js

John -- Believe me, I tried to call makeObservable in Super. That was
originally the dream.  :-)
You are correct -- calling does not work to class initialization order
(which I now understand better than ever, at least for the time being. :-))
I think using a timer would be very dangerous, and a non-starter given that
we often start reactions within constructors.
On Mon, Nov 23, 2020 at 12:29 PM John Haynes<EMAIL_ADDRESS>wrote:

We can't just put a call to makeObservable(this) in the HoistModel ctor?
Does that not work because of class initialization order or something? If
it doesn't work, could we put that call in a setTimeout or is there some
reason it absolutely needs to be in the constructor?
Just trying to brainstorm some ways to avoid requiring all models to add a
ctor and remember to make that call ... especially since it seems like it
will not be necessary forever if they would finally approve a decorators
proposal for js
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/xh/hoist-react/pull/2197#issuecomment-732309758, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AARTL2MGPX62TL7DX6F3M3DSRKLXFANCNFSM4T55JM3A
.

Agree with your point about enhancing managed, and writing destroy methods
only when necessary.
On Mon, Nov 23, 2020 at 12:34 PM Lee Wexler<EMAIL_ADDRESS>wrote:

John -- Believe me, I tried to call makeObservable in Super. That was
originally the dream.  :-)
You are correct -- calling does not work to class initialization order
(which I now understand better than ever, at least for the time being. :-))
I think using a timer would be very dangerous, and a non-starter given
that we often start reactions within constructors.
On Mon, Nov 23, 2020 at 12:29 PM John Haynes<EMAIL_ADDRESS>wrote:

We can't just put a call to makeObservable(this) in the HoistModel ctor?
Does that not work because of class initialization order or something? If
it doesn't work, could we put that call in a setTimeout or is there some
reason it absolutely needs to be in the constructor?
Just trying to brainstorm some ways to avoid requiring all models to add
a ctor and remember to make that call ... especially since it seems like it
will not be necessary forever if they would finally approve a decorators
proposal for js
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/xh/hoist-react/pull/2197#issuecomment-732309758, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AARTL2MGPX62TL7DX6F3M3DSRKLXFANCNFSM4T55JM3A
.


Milestone: Add mobile menu with java script

Added navigation menu HTML for mobile
Added navigation menu CSS for mobile
Added navigation menu javascript for mobile

Thanks, @Qoosim


MqttClient.ConnectAsync() with Tls enabled: 'RemoteCertificateChainErrors' (RevocationStatusUnknown) on Android, but not on iOS

Describe the bug
We have a HiveMQ-Broker with TLS Encryption enabled. We were able to connect via iOS, but got an error on android, even though we do not use platform-specific code. To inspect the issue, we added a debug output to the CertificateValidationHandler, like described in the client sample 'Inspect_Certificate_Validation_Errors()'. There we can see the difference between iOS and Android.
For iOS we see, that there are no errors in validating the certificate chain.
For Android we see, that there are 'RemoteCertificateChainErrors', because the RevocationStatus is unknown.
The certificate chain presented by the HiveMQ Broker is valid, we already checked that and it is reproducable on different brokers not only HiveMQ.
Which component is your bug related to?

Client

To Reproduce
Steps to reproduce the behavior:

Using this code, it's everything in there, like an example broker. Deploy it to android and iOS to get the described behavior.

`var mqttFactory = new MqttFactory();

        using (var mqttClient = mqttFactory.CreateMqttClient())
        {
            var mqttClientOptions = new MqttClientOptionsBuilder().WithTcpServer("689fc9876685493f8f11ba11d0154f35.s2.eu.hivemq.cloud", 8883)
                .WithCredentials("AppIot", "Michael1$")
                .WithTlsOptions(
                    o =>
                    {
                        o.WithCertificateValidationHandler(
                            eventArgs =>
                            {
                                System.Diagnostics.Debug.WriteLine(eventArgs.Certificate.Subject);
                                System.Diagnostics.Debug.WriteLine(eventArgs.Certificate.GetExpirationDateString());
                                System.Diagnostics.Debug.WriteLine(eventArgs.Chain.ChainPolicy.RevocationMode);
                                System.Diagnostics.Debug.WriteLine("Chain Certs");
                                eventArgs.Chain.ChainElements.ToList().ForEach(x => System.Diagnostics.Debug.WriteLine(x.Certificate.Subject));
                                System.Diagnostics.Debug.WriteLine("ChainElements Status");
                                eventArgs.Chain.ChainStatus.ToList().ForEach(x => System.Diagnostics.Debug.WriteLine($"{x.Status}: {x.StatusInformation}"));
                                System.Diagnostics.Debug.WriteLine("SSL Policy Error");
                                System.Diagnostics.Debug.WriteLine(eventArgs.SslPolicyErrors);
                                return true;
                            });
                    })
                .Build();

            // In MQTTv5 the response contains much more information.
            using (var timeout = new CancellationTokenSource(5000))
            {
                await mqttClient.ConnectAsync(mqttClientOptions, timeout.Token);
            }
        }

Expected behavior
Because we have the same connect-code and the same broker we connecting to, there should not be a different behavior on iOS and Android. Android should behave like iOS.
Screenshots
No screenshots, because all code needed is given in "To reproduce" above,
Additional context / logging
Add any other context about the problem here.
Include debugging or logging information here:
iOS debug log:
CN=*.s2.eu.hivemq.cloud
3/2/2024 11:20:23 PM
Online
Chain Certs
            CN=*.s2.eu.hivemq.cloud
            CN=R3, O=Let's Encrypt, C=US
            CN=ISRG Root X1, O=Internet Security Research Group, C=US
ChainElements Status
SSL Policy Error
            None

Android debug log:
CN=*.s2.eu.hivemq.cloud
02.03.2024 23:20:23
Online
Chain Certs
            CN=*.s2.eu.hivemq.cloud
            CN=R3, O=Let's Encrypt, C=US
            CN=ISRG Root X1, O=Internet Security Research Group, C=US
ChainElements Status
            RevocationStatusUnknown: Certificate does not specify OCSP responder
            RevocationStatusUnknown: The certificate's revocation status could not be determined.
SSL Policy Error
            RemoteCertificateChainErrors

which android version are you using? There are sone known incompatible OS with Let's Encrypt certs.
https://letsencrypt.org/docs/certificate-compatibility/
To workaround you can:

Configure the RevocationMode as NoCheck (not ideal), or
Include the CA file in your app, and use WithTrustChain

https://github.com/dotnet/MQTTnet/blob/f82d382f23ec8388ed8a33c48b7ede08e2c18622/Samples/Client/Client_Connection_Samples.cs#L452


feat: Little Improvements on pipeline view

Little improvements this time 😉

Fix #29: I enrich logs with a link element when I see something begin by http/https. The regex is very simple, but I think it's enough for the moment.

Fix #36: Adding a message while waiting (ping @vbehar)

The last one is for @jstrachan 😉  I saw that you use this app on phone. I improve a little the pipeline view on mobile/tablet.
Links to #28.
Thats looks like :

Moto G4

Pixel 2

Iphone X

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign florianorpeliere
You can assign the PR to them by writing /assign @florianorpeliere in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/approve

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: vbehar
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [vbehar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


HADOOP-16430.S3AFilesystem.delete to incrementally update s3guard with deletions

This started off as a performance improvement on very large directory trees: update the metastore table as pages of files are deleted.
But while coding it I discovered that while delete() updated S3Guard with changes, it never used it to get a consistency directory listing
That means recursive directory tree deletes could be inconsistent. This is bad. This PR

Moves the delete code into its own DeleteOperation alongside the rename one
Both of which now share the same callbacks for a limited set of methods they can invoke on the S3A FS
Uses the s3guard enabled list to scan the dir tree, skipping the use of tombstones to invalidate entries (so recovers from OOB changes)
in auth mode, does a final raw S3 scan and cleanup, so even if that is inconsistent, all is good

Test changes as appropriate for debugging things, working out why InconsistentClient wasn't being inconsistent, etc, etc

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
38
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 13 new or modified test files.

_ trunk Compile Tests _

0
mvndep
68
Maven dependency ordering for branch

+1
mvninstall
1296
trunk passed

+1
compile
1156
trunk passed

+1
checkstyle
159
trunk passed

+1
mvnsite
139
trunk passed

+1
shadedclient
1137
branch has no errors when building and testing our client artifacts.

+1
javadoc
96
trunk passed

0
spotbugs
64
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
184
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

+1
mvninstall
85
the patch passed

+1
compile
1087
the patch passed

+1
javac
1087
the patch passed

-0
checkstyle
148
root: The patch generated 9 new + 63 unchanged - 1 fixed = 72 total (was 64)

+1
mvnsite
117
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
747
patch has no errors when building and testing our client artifacts.

-1
javadoc
31
hadoop-tools_hadoop-aws generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)

+1
findbugs
224
the patch passed

_ Other Tests _

-1
unit
561
hadoop-common in the patch failed.

+1
unit
73
hadoop-aws in the patch passed.

+1
asflicense
43
The patch does not generate ASF License warnings.

7458

Reason
Tests

Failed junit tests
hadoop.fs.contract.localfs.TestLocalFSContractGetFileStatus

hadoop.fs.contract.rawlocal.TestRawlocalContractGetFileStatus

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1359

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 87a0fb805e4e 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b1eee8b

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/1/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/1/testReport/

Max. process+thread count
1468 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/1/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

OK, even with the patch about to surface we have an OOB problem with empty dirs. That is: shortcut checks on empty dirs markers preclude looking for children, so there's no scan afterwards.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
41
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 14 new or modified test files.

_ trunk Compile Tests _

0
mvndep
68
Maven dependency ordering for branch

+1
mvninstall
1078
trunk passed

+1
compile
1024
trunk passed

+1
checkstyle
138
trunk passed

+1
mvnsite
128
trunk passed

+1
shadedclient
990
branch has no errors when building and testing our client artifacts.

+1
javadoc
112
trunk passed

0
spotbugs
73
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
209
trunk passed

_ Patch Compile Tests _

0
mvndep
30
Maven dependency ordering for patch

+1
mvninstall
93
the patch passed

+1
compile
979
the patch passed

+1
javac
979
the patch passed

-0
checkstyle
151
root: The patch generated 2 new + 64 unchanged - 3 fixed = 66 total (was 67)

+1
mvnsite
128
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
720
patch has no errors when building and testing our client artifacts.

+1
javadoc
109
the patch passed

+1
findbugs
203
the patch passed

_ Other Tests _

+1
unit
553
hadoop-common in the patch passed.

+1
unit
95
hadoop-aws in the patch passed.

+1
asflicense
53
The patch does not generate ASF License warnings.

6952

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1359

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux a742a228f7c8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 872cdf4

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/2/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/2/testReport/

Max. process+thread count
1395 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/2/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

thanks. I;ve just pushed up another comment for you to look at, and I am making sure I run it without s3guard as well as with. Tested, S3 ireland
undeleted files
A ITestS3AContractRootDir run failed failing with undeleted file "fork-0005/test/testFile". That filename is used in too many tests to identify the problem.
The latest patch uses a unique name for each test case, so if the problem recurs, I can start tracking down the issue.
I don't think it's related to my changes in deletion, but given how critical delete is for cleanup, I am not ignoring it.
At the same time, I think with this patch, we are actually being more rigourous in cleanup. We use see guard to identify and delete all files, even when S3 being inconsistent. And when the client is using s3guard as an authoritative store, we still bypass it for a final bit of due diligence.

[ERROR] testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 38.181 s  <<< FAILURE!
java.lang.AssertionError:
Expected no results from listFiles(/, true), but got 1 elements:
S3ALocatedFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0005/test/testFile; isDirectory=false; length=1; replication=1; blocksize=33554432;
   modification_time=1567098043000; access_time=0; owner=stevel; group=stevel; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true;
    isErasureCoded=false}[eTag='55a54008ad1ba589aa210d2629c1df41', versionId='']
	at org.junit.Assert.fail(Assert.java:88)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.assertNoElements(AbstractContractRootDirectoryTest.java:218)
	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:200)
	at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

BTW, note in the stack how S3ALocatedFileStatus now prints etag and version? There's now lossless conversion between that and S3AFileStatus, which I needed for the isDirectory flag to not get lost, as that is used in the delete operation to choose whether to add a / on a path. This why there's a new package-private ctor for S3AFileStatus.
With the changes in, the file at fault appears to be testFailedMetadataUpdate from ITestS3AMetadataPersistenceException; doing more cleanup there. That test is deliberately creating failures in the metastore update process; maybe if the normal test FS doesn't know about the file then it's test clenanup doesn't find it. Now I know that empty dir markers will stop a scan for and delete of objects, I am starting to wonder if that is the cause of some intermittent test failures we've had in the past -though those could just have come from S3 list inconsistency not finding files to delete.
Speed
I added a section in the DeleteOperation about opportunities to speak up that process through better parallelisation.
I also make it clear that you should only derive such changing from data created in benchmarks running in ECT itself.
If you test remotely, latency can dominate, but also you are less prone to encountering throttling
on AWS services, because you are not generating enough load.
I don't immediately plan to do such performance tuning as I can see opportunities in speeding up directory listings which are on critical path for query planning. This patch here already make things faster especially very large directories. A question I have no information off is how important is directory deletion performance to applications? That's large directories. There's always going to be a fixed amount of needless overhead for deleting even a small directory: Head List delete calls, and setting up fake parent directories as needed. It's a larger directories that Dynamo DB calls come to dominate, and only once you have many thousands of files in a directory tree, that the advantages of more parallel batches of deletions probably start to deliver benefits.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
39
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 18 new or modified test files.

_ trunk Compile Tests _

0
mvndep
67
Maven dependency ordering for branch

+1
mvninstall
1070
trunk passed

+1
compile
1039
trunk passed

+1
checkstyle
141
trunk passed

+1
mvnsite
125
trunk passed

+1
shadedclient
977
branch has no errors when building and testing our client artifacts.

+1
javadoc
105
trunk passed

0
spotbugs
66
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
188
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

+1
mvninstall
81
the patch passed

+1
compile
966
the patch passed

+1
javac
966
the patch passed

-0
checkstyle
139
root: The patch generated 3 new + 93 unchanged - 5 fixed = 96 total (was 98)

+1
mvnsite
121
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
650
patch has no errors when building and testing our client artifacts.

-1
javadoc
34
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

+1
findbugs
203
the patch passed

_ Other Tests _

+1
unit
568
hadoop-common in the patch passed.

+1
unit
88
hadoop-aws in the patch passed.

+1
asflicense
44
The patch does not generate ASF License warnings.

6772

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1359

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux b829024a92bb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / c929b38

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/3/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/3/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/3/testReport/

Max. process+thread count
1397 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
135
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 18 new or modified test files.

_ trunk Compile Tests _

0
mvndep
78
Maven dependency ordering for branch

+1
mvninstall
1361
trunk passed

+1
compile
1133
trunk passed

+1
checkstyle
151
trunk passed

+1
mvnsite
144
trunk passed

+1
shadedclient
1088
branch has no errors when building and testing our client artifacts.

+1
javadoc
102
trunk passed

0
spotbugs
76
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
210
trunk passed

_ Patch Compile Tests _

0
mvndep
31
Maven dependency ordering for patch

+1
mvninstall
97
the patch passed

+1
compile
1123
the patch passed

+1
javac
1123
the patch passed

-0
checkstyle
155
root: The patch generated 3 new + 93 unchanged - 5 fixed = 96 total (was 98)

+1
mvnsite
139
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
747
patch has no errors when building and testing our client artifacts.

-1
javadoc
41
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

+1
findbugs
224
the patch passed

_ Other Tests _

+1
unit
627
hadoop-common in the patch passed.

+1
unit
101
hadoop-aws in the patch passed.

+1
asflicense
50
The patch does not generate ASF License warnings.

7833

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1359

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 1c16e674fd98 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / c929b38

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/4/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/4/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/4/testReport/

Max. process+thread count
1658 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
79
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 18 new or modified test files.

_ trunk Compile Tests _

0
mvndep
29
Maven dependency ordering for branch

-1
mvninstall
260
root in trunk failed.

-1
compile
40
root in trunk failed.

+1
checkstyle
189
trunk passed

+1
mvnsite
125
trunk passed

+1
shadedclient
1255
branch has no errors when building and testing our client artifacts.

+1
javadoc
87
trunk passed

0
spotbugs
61
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
191
trunk passed

_ Patch Compile Tests _

0
mvndep
26
Maven dependency ordering for patch

+1
mvninstall
82
the patch passed

+1
compile
1110
the patch passed

-1
javac
1110
root generated 1467 new + 1 unchanged - 0 fixed = 1468 total (was 1)

-0
checkstyle
142
root: The patch generated 3 new + 93 unchanged - 5 fixed = 96 total (was 98)

+1
mvnsite
130
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
660
patch has no errors when building and testing our client artifacts.

-1
javadoc
38
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

+1
findbugs
215
the patch passed

_ Other Tests _

+1
unit
566
hadoop-common in the patch passed.

+1
unit
113
hadoop-aws in the patch passed.

+1
asflicense
53
The patch does not generate ASF License warnings.

5440

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1359

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux a174f9a9094a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 915cbc9

Default Java
1.8.0_222

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/5/artifact/out/branch-mvninstall-root.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/5/artifact/out/branch-compile-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/5/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/5/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/5/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/5/testReport/

Max. process+thread count
1378 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Some checkstyles; will fix
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java:110:  S3AFileStatus(Path path,:3: More than 7 parameters (found 9). [ParameterNumber]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMetadataPersistenceException.java:116:                () -> { outputStream.close(); });:23: '{' at column 23 should have line break after. [LeftCurly]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardListConsistency.java:536:    Path root = path("testInconsistentS3ClientDeletes-" + DEFAULT_DELAY_KEY_SUBSTRING);: Line is longer than 80 characters (found 87). [LineLength]

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
91
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 18 new or modified test files.

_ trunk Compile Tests _

0
mvndep
75
Maven dependency ordering for branch

+1
mvninstall
1363
trunk passed

+1
compile
1329
trunk passed

+1
checkstyle
165
trunk passed

+1
mvnsite
127
trunk passed

+1
shadedclient
1126
branch has no errors when building and testing our client artifacts.

+1
javadoc
118
trunk passed

0
spotbugs
81
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
225
trunk passed

_ Patch Compile Tests _

0
mvndep
26
Maven dependency ordering for patch

+1
mvninstall
95
the patch passed

+1
compile
1222
the patch passed

+1
javac
1222
the patch passed

-0
checkstyle
179
root: The patch generated 2 new + 93 unchanged - 5 fixed = 95 total (was 98)

+1
mvnsite
126
the patch passed

+1
whitespace
1
The patch has no whitespace issues.

+1
shadedclient
770
patch has no errors when building and testing our client artifacts.

-1
javadoc
30
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

+1
findbugs
206
the patch passed

_ Other Tests _

+1
unit
562
hadoop-common in the patch passed.

+1
unit
98
hadoop-aws in the patch passed.

+1
asflicense
48
The patch does not generate ASF License warnings.

8059

Subsystem
Report/Notes

Docker
Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1359

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux be7f78b51008 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 1ae7759

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/6/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/6/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/6/testReport/

Max. process+thread count
1343 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/6/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
42
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 18 new or modified test files.

_ trunk Compile Tests _

0
mvndep
25
Maven dependency ordering for branch

+1
mvninstall
1058
trunk passed

+1
compile
1016
trunk passed

+1
checkstyle
140
trunk passed

+1
mvnsite
129
trunk passed

+1
shadedclient
989
branch has no errors when building and testing our client artifacts.

+1
javadoc
106
trunk passed

0
spotbugs
68
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
188
trunk passed

_ Patch Compile Tests _

0
mvndep
24
Maven dependency ordering for patch

+1
mvninstall
81
the patch passed

+1
compile
973
the patch passed

+1
javac
973
the patch passed

-0
checkstyle
148
root: The patch generated 2 new + 93 unchanged - 5 fixed = 95 total (was 98)

+1
mvnsite
149
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
711
patch has no errors when building and testing our client artifacts.

-1
javadoc
39
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

+1
findbugs
203
the patch passed

_ Other Tests _

+1
unit
549
hadoop-common in the patch passed.

+1
unit
99
hadoop-aws in the patch passed.

+1
asflicense
51
The patch does not generate ASF License warnings.

6835

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1359

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 1c11fbfaeee9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 337e9b7

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/7/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/7/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/7/testReport/

Max. process+thread count
1570 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/7/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Thanks for the vote, merging in. Your reviews are always valued, and when you feel the urge to start coding again,...

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
10
https://github.com/apache/hadoop/pull/1359 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/1359

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1359/8/console

versions
git=2.17.1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

This is merged. Thanks


Respect keyForRelationship for embedded records

fixes #372

I added some code to handle removing hasMany when keyForRelationship is overwritten. This was not working previously as the record cache was accessing the wrong payload key.

Thanks for this, sorry it took so long for me to properly look over it.
looks good, are you still good to merge it?

Yep, it's good to go.


feat: maybe and either

Hi, thank you for maintaining this package !
This is an attempt to implement #4 : Maybe and Either

Thanks! It's a library that I use personally for playing around with Deno. :P
I'll take a look at the PR after work but from what I have seen it looks pretty nice.
CI matrix tests for Deno 0.32.0 and 0.31.0 fail because Deno.test was introduced in 0.33.0. I have to fix that. 🤔

Hey!  I think these are great and I think they'll be a great fit. There are a couple of small things that I was thinking about adding (now or later as separate issues).
One is the usage of [Deno.symbols.customInspect] for easier debugging, especially for people less accustomed to functional programming and who want to just console.log stuff and see what it is.
For example in Maybe:
// ...
mapMaybe<R>(f: (t: T) => Maybe<R>): Maybe<R>;

    toJSON(): T | undefined;
    toString(): string;
    [Deno.symbols.customInspect]: () => string;
}

And then in just:
// ...
*[Symbol.iterator]() {
    yield value;
},
[Deno.symbols.customInspect]() {
    return `just{${value}}`
}

Another thing that I noticed (but that's something that we probably won't be able to overcome in current TypeScript) is that in some cases .map on Maybe will lose typings and default to unknown so <number> cast (for example) is required.
const double = (x: number) => x * 2;

const input: any = [5, 2, 1, 5, 7];

const res = maybe(input)
    .map(reverse)
    .map<number>(head)
    .map(double)
    .default(0);

Lastly, there are some minor README duplications that I will mark in the code. :)

@galkowskit is this ok for you ?

Yup, great job! Looks good to me, I'll squash and merge it. And I'll fix the CI this evening so it doesn't fail on 0.32.0. :)

🎉


How to get screen name directly from full url ?

I am looping on many url and need to get screen name only from url as :
https://twitter.com/AlArabiya
I need to get AlArabiya
How that ?

you can use simply jquery
var url = $('#posteng_iddd').val();
var urlid = url.substring(url.lastIndexOf("/") + 1);

@msliman If you are getting data as the object you can directly do get screen_name from $post->screen_name. If you are getting data in array format then $post['screen_name']. You have to set up response type in the request it's self. Above syntax can be used for the PHP, if you are using another language then follow that syntax. You can get screen_name from post data, you don't need to do URL's string processing to retrieve screen_name. Go to this link for reference. First, check that you are at the correct hierarchical level of array or object. Most of the time data is many levels deeper in response.

@HARIKESHKUMAR String processing is totally wrong because you are getting data in response, why anyone would do string processing if you already have data in responses(Object/Array).

Closing this due to inactivity. Please let me know if you are still having an issue and i'll reopen.


Better failover suitability, promisified. And lock extension.

You might be interested. If not, feel free to close.
I needed a distributed locking solution that provides a bit more consistency in a failover/sentinel environment than, well pretty much any Redis-based locking solution I could find. So I decided to take yours as a starting point. The key here is mainly to implement acquireLock as a pipeline that includes a wait. New config parameters minReplications and replicationTimeout control this, by default it's disabled.
Also

Promisified the API, so that it fits in our own codebase (but provided callback interfaces for backwards compatibility)
Added an extendLock method that extends the TTL of an existing lock
Added some tests, including one to test expiry (so useful regardless of my other additions)

Thanks! I can't believe I did not know about the Redis WAIT command so far 🤔

It doesn't give 100% guarantees (except when you set it to await replication to all replicas I suppose, but then it would also become unavailable when one of the replicas is unavailable) but in my tests without the wait it was pretty much guaranteed that confirmed locks would be lost in case of a failover. Even when it was triggered by a manual 'failover' command on a sentinel (ie, no unanticipated loss of primary). I haven't been able to reproduce lock loss in my (admittedly quite primitive) tests with the 'wait', even when it's set to just 1.
However note that it will become unavailable (locking will fail, but at least you will know) for usually a few seconds during a failover.
I guess it could be improved by silently retrying the lock if there is a replication failure within the replication timeout (which will happen if the primary is lost while waiting for replication). That way such unavailability events would become unnoticeable (apart from the longer delay) for applications that set sufficiently long timeouts.
You may also consider just using the lock timeout for the replication timeout.

BTW, maybe remove the extra 'originally forked from...' line.

Same for the altered npm install instructions. Apologies, I should have kept those out of the pull request. #2 will fix that.


NLP-ENGINE-386 Added reload KB flag and logic

@amnonma for review

@amnonma for some reason, this is not working. Something is not set correctly after deleting.

How do you set and test this.  Is there a button or menu item somewhere?
@dehilsterlexis


[0.7-preview] Can not create guest vm from ISO image with iSCSI on Btrfs

I just test zstack 0.7-preview version, and I try to create guest vm from ISO image but it fails:
{"org.zstack.header.vm.APICreateVmInstanceMsg":{"name":"VM1","description":null,"instanceOfferingUuid":"8dc33e6ae39a45f3968b3170185cfea5","imageUuid":"50952352b1fe483b9b9ad07370739da2","l3NetworkUuids":["eeb9feff031243f2ae4928c3f3882ad6"],"rootDiskOfferingUuid":"aa995c8e171b4e39aa3e7b7b85ccd18e","dataDiskOfferingUuids":[],"zoneUuid":null,"clusterUuid":null,"hostUuid":null,"resourceUuid":"b1d10ea8b02b40818686768abe42e196","defaultL3NetworkUuid":"eeb9feff031243f2ae4928c3f3882ad6","systemTags":["hostname::vm1"],"session":{"uuid":"b3000cee427549d494ed0b706a3ea219"}}}
the resault is :
{"org.zstack.header.vm.APICreateVmInstanceEvent":{"success":false,"apiId":"1394b85890964031aca6d7e6a084b8e5","headers":{"schema":{}},"error":{"code":"SYS.1006","cause":{"code":"SYS.1000","description":"An internal error happened in system","details":"unhandled exception happened when calling private void org.zstack.kvm.KVMHost.startVm(org.zstack.header.vm.VmInstanceSpec, org.zstack.header.message.NeedReplyMessage, org.zstack.header.core.NoErrorCompletion), null"},"description":"An operation failed","details":"unhandled exception happened when calling private void org.zstack.kvm.KVMHost.startVm(org.zstack.header.vm.VmInstanceSpec, org.zstack.header.message.NeedReplyMessage, org.zstack.header.core.NoErrorCompletion), null"},"id":"062aa8552c7b481aa088cb33d6369aeb","creatingTime":1430834985652}}

fixed in mainline


fix: features

:tada: This PR is included in version 0.122.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


No rewards received via BTC-DFI LM

In the DEX I had swapped 5 DFI to BTC. After successful swap I had deposited the DFI (4,99) and BTC (0,00011303) to the BTC-DFI pool on Nov. 30 20:00 UTC. Today, about 24h later I can still not see any received rewards in my BTC-DFI wallet. Pool share with this token amount of course is very small (currently 0,0004%).
Blockchain shows up as fully sychronized. App version is 2.1 on Windows 10 Pro with latest OS updates.
Note: The app had not been freshly installed but rather updated from previous version 2.0.1. Prior to the above mentioned DEX transaction and pool deposit the blockchain had also been fully synced.
Note: While writing this bug entry the blockchain synchronization status suddenly changed from "fully synced" back to 71,63% (Block 472451) and then again jumped back to "fully synced".

defichain_app_2.1_applogs.log
defichain_app_2.1_binlogs.log

Hi @stefanowitschdev , thanks for your feedback. The team is currently addressing the "no rewards received" issue, more details will be stated in the upcoming update. Regarding the synchronising NaN% issue, it has been fixed in release v2.1.1 https://github.com/DeFiCh/app/releases/tag/v2.1.1.


Fix to work with rollup-plugin-typescript2

Resolve #127

@neoziro thanks for checking my PR! Now I've fixed common .babelrc problem.

Thanks! I am sorry maybe I wasn't clear in my explanations. A .babelrc impact the build, in this specific case we need to specify the Babel configuration directly in the code. In the call of babel transform just here https://github.com/smooth-code/svgr/pull/147/files#diff-8fab4aab82ae41a7c435ac29a5b56e25R59

Sorry for my miss understood. Your suggestion changes result of source code from SVG right? However, What this PR needed is remove async/await syntax from packages/rollup/lib/index.js actually. Because rollup-plugin-typescript-2 uses object-hash to create cache directory (key) with rollup config file at here. Then  problem is the object-hash. This doesn't able to handle async function syntax. Issue is here: Support async functions.
Ways to solve this is that

Fix object-hash to support async/await syntax
Avoid to use object-hash from rollup-plugin-typescript-2
Provide rollup-plugin without async/await (This PR)

Because 1 or 2 seems take quite a while, Is it possible to provide rollup plugin with condition 3 until 1 or 2 fixes?


Bump dependencies

Fixes vulnerabilities around lodash and mixin-deep/set-value/union-deep.
Changelog Entry

*: Bumps lodash-*(https://www.npmjs.com/package/lodash), by @compulim, in PR #2199

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Description
Result of npm audit fix.
Specific Changes

Updated bundle/component/core
Updated playground and other samples that is based on create-react-app

[ ] Testing Added

Coverage remained the same at 64.541% when pulling fefa5ba85e27efdbbb2c8deaede7b6958b7c6c83 on compulim:bump-lodash into 83edbd04c0d3aecd6b27e4e396c7c7c45cb3350c on microsoft:master.


[Feature] Add phone to stg_zendesk__user

Is there an existing feature request for this?

[X] I have searched the existing issues

Describe the Feature
Looks like the staging models are a bit out of date with what fields fivetran actually replicates from Zendesk. This is just one I've noticed that would be helpful, but there are others.
Describe alternatives you've considered
No response
Are you interested in contributing this feature?

[ ] Yes.
[ ] Yes, but I will need assistance and will schedule time during your office hours for guidance.
[ ] No.

Anything else?
No response

hi @trunsky! Yeah phone definitely makes sense to include. Curious to hear about the other missing fields you think would be helpful (in user or other tables)
Also, if you're using the zendesk transform package, do you envision phone being passed to any end models there? Or is just having it in the staging user model sufficient?

Hello @fivetran-jamie! I just started using it yesterday to take advantage of the SLA model. We had some existing reporting on tickets I was going to cut over to using it as well, and that is just one field I noticed was missing. For us, it would be helpful on the ticket related models.

Cool good to know!
For the time being, I am going to leave this feature request open to allow other members of the community to chime in if they would find this field helpful. If more community members engage we will happily accept this feature into an upcoming sprint.

hey @trunsky 👋 included this update in the most recent release, v0.11.1
going to close this issue, but i know you mentioned potentially adding other fields as well. please feel free to open another issue if there are specific ones you'd want to see added too!


Fix IllegalStateException in ServletAsyncContextDelegateProvider

Add a comment in the commit message for the second one explaining why we're switching

looks good


Fix zcompile error with zsh 5.4

Error is:
$ zcompile highlighters/main/test-data/function.zsh
zsh: defining function based on alias `ls'
zsh: parse error near `()'
zcompile: can't read file: highlighters/main/test-data/function.zsh

View Incompatibilities since 5.3.1 in https://github.com/zsh-users/zsh/blob/bb218704d27bcca9aa4426296dcd5c13d58b330a/README

I'm aware of the upstream change, thanks, see 9523d6d49cb3d4db5bd84c3cec6168a2057fe3ab.
What I don't understand is why this is needed in the first place.  z-sy-h does not alias ls anywhere, and in fact, it goes into some trouble to ensure that it won't be affected by any alias ls=... that .zshrc might be doing: https://github.com/zsh-users/zsh-syntax-highlighting/blob/4cde79ef0026f9b543d2beffc90d6199239dbfa0/zsh-syntax-highlighting.zsh#L30-L36
And separately, what's the use-case for zcompiling test-data files (!)?

@danielshahaf, good points.
Should we change the test to do only:
BUFFER='cd;ls'

expected_region_highlight=(
  "1 2 builtin" # cd
  "4 5 command" # ls
)

without the functions? There's also a comment at the end of the test I quite don't understand:
# ... cd() and ls() should still be a functions
# when _zsh_highlight runs.  Leaving the wrapper functions is harmless.

And it makes no sense to compile the test-data. I'm just compiling everything under highlighters. I should fix that on my side...

No, we shouldn't.  The whole point of the test is to test that a function that shadows a builtin (or a command) is highlighted as a function.
I've fixed the comment in 5436d3e5fc1e, thanks for pointing it out.
Does this address everything?

Does this address everything?

Isn't function ls preferred over ls(), to address the issue when ls is already an alias...?

Aliases typically change the default behaviour (e.g., alias ls='ls -F') in ways that may break scripts.  The right answer is for .zshrc aliases not to be in effect while running the test suite.
That's already the case when the test suite is launched by make test.
See also #444.

Aliases typically change the default behaviour (e.g., alias ls='ls -F') in ways that may break scripts.

To be clear here, I refer to scripts that run ls, e.g., if ls | grep '^foo$'; then …, with no functions involved.

Sounds good. Closing this. Not compiling the files under test-data works for me, and is also an optimization. Thank you!

You're welcome.  Thanks for the report!


Support cast run quorum private txn

This PR is raised to support debugging Quorum private transactions with cast run. Context: https://github.com/foundry-rs/foundry/issues/9057
Motivation
Quorum private transactions are not generating the right traces with cast tooling due to the way that the client masks the actual transaction input data.
Solution
Quorum private transactions have a specified format that is unique to the blockchain. The transaction input is a 64 bytes Tessera hash, and the recovery id of a private transaction can only be 37/38. Based on the criteria, we can get the CLI to make an additional RPC call to fetch the actual private transaction input data to replace the tessera hash.

Personally not a fan as it is highly vendor specific and therefore I would be hesitant to add this; not because of the proposed functionality of implementation but whenever we implement a feature in Foundry we try to be as neutral as possible, avoid having vendor-specific logic and avoid enshrining winners.
A good example of how a vendor specific request for Kontrol cheatcodes was turned into generalized functionality available to all and non-specific can be seen here: https://github.com/foundry-rs/foundry/issues/4072


Doors

About the PR
Adds doors:

[x] Airlock assembly (sprites and prototype)
[x] Personal door (sprites and prototype)
[x] Personal glass door  (sprites and prototype)
[x] Maintenance door  (sprites and prototype)
[x] Command door (sprites and prototype)
[x] Medical door (sprites and prototype)
[x] Medical glass door (sprites and prototype)
[x] Engineering door (sprites and prototype)
[x] Engineering glass door (sprites and prototype)
[x] Security door (sprites and prototype)
[x] Security glass door (sprites and prototype)
[x] Add accesses to the airlocks.

Why / Balance
Resolves #590
Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

will do windoors in the different PR.

Ready to review.

also for the note,access configurator and painter does not works on cm airlocks.

also for the note, access configurator and painter does not works on cm airlocks.

Shouldnt be needed

double doors are also missing


[Mini.starter] issue with 'sh' files directly from the terminal - notification error

Contributing guidelines

[X] I have read CONTRIBUTING.md
[X] I have read CODE_OF_CONDUCT.md
[X] I have updated 'mini.nvim' to latest version

Module(s)
Mini.starter
Description
I'd like to report an issue with Mini.starter. When I attempt to open bash/zsh/sh formatted files directly using the nvim .example command, I encounter an error. The error message initially prompts me to "Press enter or type a command to continue."
After pressing enter, I receive the following error: (mini.starter) 'buf_id' in 'refresh()' is not a valid Starter buffer identifier.
Please note that I observed a similar issue mentioned in #230, but the steps to reproduce it differ.
Neovim version
0.9.2
Steps to reproduce

create any hidden file without file ext like (.zshrc / .bzshrc / .examplerc)
copy this line to the file: #!/bin/zsh to tell it its sh format and write then close nvim.
run this command in terminal to open the file nvim .examplerc
the issue would appear

Expected behavior
No response
Actual behavior
A notification is sent by mini.starter saying "buf_id in refresh() is not an identifier of valid Starter buffer". This does not occur with the any other file or if you open the file .zshrc after opening neovim.

Thanks for the issue!
I can not reproduce. Neither with '.examplerc' from the issue, nor from mine '.zshrc'. Also neither with Neovim 0.9.2, nor with current Nightly.
Could you try to reproduce this with only 'mini.starter'?
Also, do you happen to have any plugins installed affecting either startup or shell files (if this happens only for shell files)?

Thank you for your quick response. I've already disabled all the plugins, but unfortunately, the issue remains unchanged.
However, when I configured another Neovim instance with minimal settings using mini.starter, the problem disappeared.
It seems the issue is related to my configuration. Therefore, I'll go ahead and close this matter. thanks again


Refactor Metadata-related code to remove dbClient global variable.

Fix #2019

Codecov Report

Merging #2132 into master will decrease coverage by 0.4%.
The diff coverage is 12.05%.

@@            Coverage Diff            @@
##           master   #2132      +/-   ##
=========================================
- Coverage    41.5%   41.1%   -0.41%
=========================================
  Files         182     182
  Lines       12258   12386     +128
=========================================
+ Hits         5088    5091       +3
- Misses       6873    6998     +125
  Partials      297     297

Impacted Files
Coverage Δ

internal/core/metadata/init.go
0% <ø> (ø)
:arrow_up:

internal/core/metadata/rest_devicereport.go
0% <0%> (ø)
:arrow_up:

internal/core/metadata/rest_provisionwatcher.go
0% <0%> (ø)
:arrow_up:

internal/core/metadata/router.go
0% <0%> (ø)
:arrow_up:

internal/core/metadata/rest_command.go
100% <100%> (ø)
:arrow_up:

internal/core/metadata/rest_deviceprofile.go
89.89% <100%> (ø)
:arrow_up:

internal/core/metadata/rest_addressable.go
93.37% <100%> (+0.11%)
:arrow_up:

internal/core/metadata/rest_device.go
1.58% <2.08%> (ø)
:arrow_up:

internal/core/metadata/rest_deviceservice.go
33.24% <29.03%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4291cdd...99556ff. Read the comment docs.


Implement historyQ.UpdateClaimableBalance

Now that we have state processing for sponsorship related changes we can implement UpdateClaimableBalance in ClaimableBalancesProcessor

Fixed in #2919


Add 'fromBigInt'

This constructor is needed for large integer values.

Looks good! Can we refactor fromInt to use this?

@hdgarrood oh of course! Done :)


Fix for astroid 2.5

Summary
Fixes #1532
Details and comments

Unit test fails due to #1534

It look like pylint 2.6.2 fixed all the problems by pinning itself astroid. A full github actions ran against current code and it worked without the pylint comments in this PR. It seems to me we don't need this PR anymore.

It is not required anymore because pylint 2.6.2 fixes the dependency. I see the results of https://github.com/Qiskit/qiskit-aqua/pull/1522


btfgen: add type based relocations support

btfgen: add type based relocations support

improve Makefile to support a debug binary

remove "poisoned" detection: detecting poisoned relocations won't
always work because sometimes, in case we're dealing with flavored btf
types, they might be placed in dead branches and, still, resulting
BTF is valid.

type based relocations: add all types for a given complex btf type: we
cannot know for sure which members/fields will be accessed, at least
not from the relo info given for a type based relocation. adding all
types won't hurt in this case as type checks are not even close to the
number of field relocations.

Here is an example on how this works:
union kernfs_node_id {
    struct {
        u32     ino;
        u32     generation;
    };
    u64         id;
} __attribute__((preserve_access_index));

struct kernfs_node___old {
    const char            *name;
    union kernfs_node_id  id;
} __attribute__((preserve_access_index));

...

static __always_inline const u64 get_cgroup_id(struct cgroup *cgrp)
{
    struct kernfs_node *kn = READ_KERN(cgrp->kn);

    if (kn == NULL)
        return 0;

    // id was not always of type u64 (but "union kernfs_node_id" before 5.5)
    // yet we can read it as u64 as the size of both types is the same
    u64 id;

    if (bpf_core_type_exists(union kernfs_node_id)) {
        // recast pointer to capture kernfs_node___old type for compiler
        struct kernfs_node___old *kn_old = (void *)kn;

        // now use old "id", which is of type "union kernfs_node_id"
        bpf_core_read(&id, sizeof(u64), &kn_old->id);
    } else {
        bpf_core_read(&id, sizeof(u64), &kn->id);
    }

    return id;
}

We, then, have 2 situations. Running in a kernel < 5.5, when kernfs_node_id will be an enum, and running in a kernel >= 5.5, when kernfs_node->id is an u64 (instead of a pointer to an enum).
By using the proposed code, the generated BTF in both situations are:
kernels < 5.5:
[74] STRUCT 'kernfs_node' size=128 vlen=1
        'id' type_id=68 bits_offset=832

[68] UNION 'kernfs_node_id' size=8 vlen=2
        '(anon)' type_id=127 bits_offset=0
        'id' type_id=40 bits_offset=0

[40] TYPEDEF 'u64' type_id=12

[127] STRUCT '(anon)' size=8 vlen=2
        'ino' type_id=1 bits_offset=0
        'generation' type_id=1 bits_offset=32

[1] TYPEDEF 'u32' type_id=32

[32] TYPEDEF '__u32' type_id=81

[81] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)

kernels >= 5.5:
[63] STRUCT 'kernfs_node' size=128 vlen=1
        'id' type_id=86 bits_offset=832

[86] TYPEDEF 'u64' type_id=92

[92] TYPEDEF '__u64' type_id=35

[35] INT 'long long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)

Tested this, works as expected. Good job @rafaeldtinoco

type based relocations: add all types for a given complex btf type: we
cannot know for sure which members/fields will be accessed, at least
not from the relo info given for a type based relocation. adding all
types won't hurt in this case as type checks are not even close to the
number of field relocations.

Actually in this case we need to add all types as you're doing in the PR. BPF_TYPE_SIZE (bpf_core_type_size) needs the complete type. Even if BPF_TYPE_EXISTS (bpf_core_type_exists) doesn't require the full type definition (structs and unions are not checked recursively but only by name) this could change in the future so better to be on the safe side.
I need to do another pass on it.
Btw, do you know why the diff is so difficult to understand in this case? It shows like you added btf_reloc_get_type and bpf_reloc_type_add_member but those are the same functions we already had.

Btw, do you know why the diff is so difficult to understand in this case? It shows like you added btf_reloc_get_type and bpf_reloc_type_add_member but those are the same functions we already had.

I think there was a code repositioning that made that happen. Sorry for that.

Some blocking comments on the array handling.
I tried the an ebpf program with the following:
	if (bpf_core_type_exists(struct task_struct)) {
		bpf_printk("type exists\n");
	}

After fixing the array type handling and the unsupported kind: void (0) errors, now I get a stack overflow. Are we missing a return on somewhere in btf_reloc_put_type_all()?

checking...

Some blocking comments on the array handling.
I tried the an ebpf program with the following:
	if (bpf_core_type_exists(struct task_struct)) {
		bpf_printk("type exists\n");
	}

After fixing the array type handling and the unsupported kind: void (0) errors, now I get a stack overflow. Are we missing a return on somewhere in btf_reloc_put_type_all()?

checking...

There is a recursive logic adding BTF_KIND_STRUCT member (of typedef) to a BTF_KIND_TYPEDEF (of the struct):
LBTF: ./btfs/5.4.0-87-generic.btf
OBJ : /home/rafaeldtinoco/work/sources/ebpf/portablebpf/example.bpf.o
AddressSanitizer:DEADLYSIGNAL
=================================================================
==2788259==ERROR: AddressSanitizer: stack-overflow on address 0x7ffff6445ff8 (pc 0x558666cf9ad2 bp 0x6060000000e0 sp 0x7ffff6446000 T0)
    #0 0x558666cf9ad2 in hashmap__find /home/rafaeldtinoco/work/sources/ebpf/btfgen/libbpf/src/hashmap.c:207
    #1 0x558666cd99ac in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:370
    #2 0x558666cd9e34 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:418
    #3 0x558666cd9cd9 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:398
    #4 0x558666cd9e34 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:418
    #5 0x558666cd9cd9 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:398
    #6 0x558666cd9e34 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:418
    #7 0x558666cd9cd9 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:398
    #8 0x558666cd9e34 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:418
    #9 0x558666cd9cd9 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:398
    #10 0x558666cd9e34 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:418
    #11 0x558666cd9cd9 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:398
    #12 0x558666cd9e34 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:418
    #13 0x558666cd9cd9 in btf_reloc_put_type_all /home/rafaeldtinoco/work/sources/ebpf/btfgen/btfgen.c:398

I'll have to check of type exists before calling the btf_reloc_put_type_all() recursively. I'll fix that.

I believe I got rid of loops within btf_reloc_put_type_all() by placing a bool that explicitly checks if the type was already manipulated by btf_reloc_put_type_all. This allows both btf_reloc_put_type() and btf_reloc_put_type_all() to co-exist, and, yet the former to add all members & fields of a given btf type.

Now I'm getting another error :(.
LBTF: ../input/5.4.0-88-generic.btf
OBJ : /home/mvb/kinvolk/ebpf/libbpf-bootstrap/examples/c/.output/minimal.bpf.o
unsupported kind (all): func_proto (17009)
ERR : failed to generate btf for ../output/5.4.0-88-generic.btf

=================================================================
==409560==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 32816 byte(s) in 1 object(s) allocated from:
    #0 0x7f1ca657abc8 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f1ca6324098 in opendir (/lib/x86_64-linux-gnu/libc.so.6+0xe1098)

SUMMARY: AddressSanitizer: 32816 byte(s) leaked in 1 allocation(s).

I'm trying with 5.4.0-88-generic.btf as the input file.

Yes, we don't support BTF_KIND_FUNC nor BTF_KIND_FUNC_PROTO types for both btf_reloc_put_type() and btf_reloc_put_type_all(). Is this because of the BTF type check for task_struct you had as example ? (I believe so, because I also faced these errors when testing with that example).
I can, for now, ignore both types and allow the logic to continue. I'm not sure how those kinds should be handled also. Do you need those BTF types for the projects you're dealing with ?

Do you need those BTF types for the projects you're dealing with ?

I don't. I just want to handle as many as possible cases to avoid problems in the future.


Add a pattern matching search and replace framework

An awesome feature to have which would make mapping, and many peep-hole compiler optimizations.
The interface would be something along the lines of:
patternMatch(Module* pattern, Module* replacement, string startinst)
This would go through all instances and see if it matches the pattern defined in pattern.def starting with the pattern.startinst.
It will then go and replace all of the matches
Implications:
This would make it very easy to write fusion passes

Initial version implemented by #84

This should actually be implemented as a legitimate pass.

This was finished in #108


[ADAM-1490] Add asSingleFile to saveAsFastq and related.

Fixes #1490

Coverage decreased (-0.03%) to 84.217% when pulling 8e4fee8256adad30b5ecb7f0b7e4d9361f3fd7b1 on heuermh:single-file-save into 640c44bae1841217f981899c3a2b793c7ae06dcd on bigdatagenomics:master.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2263/
Test PASSed.

Coverage decreased (-0.7%) to 83.524% when pulling fb879805970eed3193779bf92c0b83053ac64ad5 on heuermh:single-file-save into 640c44bae1841217f981899c3a2b793c7ae06dcd on bigdatagenomics:master.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2272/
Test PASSed.

Merged! Thanks @heuermh!


[5.1][sourcekitd] Improve CursorInfo, RelatedIdents, and local rename to better handle VarDecls involved in case fallthroughs

Cherry-pick of https://github.com/apple/swift/pull/23448 for 5.1 branch

@swift-ci test

Build failed
Swift Test Linux Platform
Git Sha - 50f6ddb46658cb19b1840730b0e6e1a275eb2076

Linux failure seems unrelated:
16:04:31 Failing Tests (1):
16:04:31     LLDB :: Expr/TestIRMemoryMap.test

@swift-ci please test Linux


Add binding annotation to record an action path not resolved

Description

If the user invokes an action in a package binding, they can't know a package binding path from which action is invoked. because the current activation structure only records the resolved action path.
for example, suppose we have the following entities:

package : pkg1

action : pkg1/action1

package pkg2 bind pkg1
package pkg3 bind pkg1

If user invokes the pkg2/action1 action and the pkg3/action1 action, all path annotations of invoked action in binding packages are saved as namespace/pkg1/action1.
    "annotations": [
        {
            "key": "path",
            "value": "namespace/pkg1/action1"
        },
     ]

Suggestion
So i added an annotation to distinguish between resolved action path and invoked action path and opend this PR.
{
    "activationId": "f4250bb2721d4e4ca50bb2721d2e4cdd",
    "annotations": [
        {
            "key": "path",
            "value": "namespace/pkg1/action1"
        },
        {
            "key": "originPath",
            "value": "namespace/pkg2/action1"
        }
    ]
}

Related issue and scope

[ ] I opened an issue to propose and discuss this change (#????)

My changes affect the following components

[ ] API
[x] Controller
[ ] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[ ] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[ ] Tests
[ ] Deployment
[ ] CLI
[ ] General tooling
[ ] Documentation

Types of changes

[ ] Bug fix (generally a non-breaking change which closes an issue).
[x] Enhancement or new feature (adds new functionality).
[ ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[x] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[x] I updated the documentation where necessary.

@rabbah
Is it possible to include this change? or any other direction is being considered?
In any case, I hope this issue is addressed so that we can resolve many binding related issues.

Thanks for the nudge. Will get on it.

conflict is resolved

Any update on this PR?

Sorry. I should resolve a conflict. I can handle today.

@rabbah  A conflict is resolved. could you please review again?
c.c. @style95

@rabbah Do you have any comments on this?

I would merge this in 72 hours if there is no objection based on silent consent.


⚠ Account has degraded performance

In fdbfebb, Account (https://account.wrk.com/health) experienced degraded performance:

HTTP code: 200
Response time: 200 ms

Resolved: Account performance has improved in 749d0ef.


Commit Gitlab  file size limit

Preflight Checklist

[x] I agree to follow the Code of Conduct that this project adheres to.
[x] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Describe the bug
Make a big diagram and save it to gitlab
In this diagram i've a lot of shape with base64 image
To Reproduce
Steps to reproduce the behavior:

Go to File
Click on Save
Click on Commit wirh a message
See error

Error saving file
Drawing is too large (985.7 kB / 1 MB)
Expected behavior
No limit in size for this kind of save.
Screenshots

draw.io version (In the Help->About menu of the draw.io editor):

draw.io version [e.g. 12.8.5] (SELF-HOSTED docker)

Desktop:

OS: windows
Browser : firefox
Version : 73.0.1

Additional context
Add any other context about the problem here.

OK ... SO i've succesfully change this limitation
This limitation comes from GitLabClient.js  line 561
-> 	if (data.length >= this.maxFileSize)
But from my point of wiew there is no reason for this .
The variable maxFileSize is an extend from GitHubClient.js
I haven't seen any where this kind of limitation in both Git .
Best Regards

Hey @jbperrin88 ,
I was looking for a solution and I found your post.
Actually I did it through my DockerFile to apply the edit, but it doesn't seem to work, and still display the error when I try to commit the diagramm on gitlab.
I tried the followed Dockerfile, could you have a look ?
FROM dockerproxy-iva.si.francetelecom.fr/jgraph/drawio:12.9.6-alpine

LABEL name="drawio-image"

# Give writable permission
RUN chmod a+rwx $CATALINA_HOME/webapps/draw

# Copy modified files to deactivate cloud storage and git size limitation
COPY PreConfig.js $CATALINA_HOME/webapps/draw/js/PreConfig.js
COPY GitHubClient.js $CATALINA_HOME/webapps/draw/js/diagramly/GitHubClient.js

WORKDIR $CATALINA_HOME

EXPOSE 8080 8443

ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["catalina.sh", "run"]

HEALTHCHECK NONE
USER 1001

In my GitHubClient.js :
GitHubClient.prototype.maxFileSize = 10000000 /*10MB*/;
in a nutshell :
Copy my own PreConfig.js though the dockerfile works
But not the GitHubClient.js

Hello
Happened for me today as well,
Drawing is too large (1016 kB / 1 MB) was the error somewhere.
I've had to resort to device / local saves.

I was able to hack a fix, but I think this should be investigated further. The reason why it does not work when you change GitLabClient.js is because the code overall is minified and needs to change app.min.js. So you either need to regenerate app.min.js after editing GitLabClient.js OR in my case I just edited app.min.js. I found the GitLabClient line to add it to, and then append it to the end of the line.  My example is below.
The real fix should be perhaps to consider making maxFileSize configurable for each of the file modes or, as jbperrin88 pointed out, perhaps we do not need a maxFileSize variable for GitLab at all. In our case, we are managing the magnitude of the file size with Git LFS in our repository rather than needing Draw.io to manage it for us.
My example is on line 10026 of app.min.js for my particular version of this example (see the end of the line for my part to set maxFileSize):
GitLabFile.prototype.getMode=function(){return App.MODE_GITLAB};GitLabFile.prototype.getDescriptorEtag=function(a){return a.last_commit_id};GitLabFile.prototype.setDescriptorEtag=function(a,e){a.last_commit_id=e};GitLabLibrary=function(a,e,c){GitLabFile.call(this,a,e,c)};mxUtils.extend(GitLabLibrary,GitLabFile);GitLabLibrary.prototype.doSave=function(a,e,c){this.saveFile(a,!1,e,c)};GitLabLibrary.prototype.open=function(){};GitLabClient=function(a){GitHubClient.call(this,a,"gitlabauth")};mxUtils.extend(GitLabClient,GitHubClient);GitLabClient.prototype.clientId=DRAWIO_GITLAB_ID;GitLabClient.prototype.scope="api%20read_repository%20write_repository";GitLabClient.prototype.baseUrl=DRAWIO_GITLAB_URL+"/api/v4";GitLabClient.prototype.authToken="Bearer";GitLabClient.prototype.maxFileSize=1E8;

I'm experiencing the same issue with GitHub saves.

Increased to 10MB for next build.

Deployed in 13.6.0

Dear, possible to extend it for GitHubClient too ?

Dear, possible to extend it for GitHubClient too ?

Please file a new issue for new requests.

Please file a new issue for new requests.

I am on v14.6 but I still see the same issue - 1 MB limit

@indrex what URL are you using? That message is 10MB in the code since 13.6. Are you sure you are saving to Gitlab?

Sorry, my bad, I was committing to a GitHub repo not to GitLab :(


Explain CodeEditor parameters within docstring #10124

Description of Changes

[x] Wrote at least one-line docstrings (for existing functions)

Added docstring explaining the parameters of setup_editor() function in codeeditor.py
Issue(s) Resolved

Fixes #10124
Affirmation
By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

I certify the above statement is true and correct:

Hello @Akashtyagi08! Thanks for opening this PR.  Where am i doing wrong ?
I am quite new to open source contribution.

Hi @Akashtyagi08 don't worry about the failing test, they are unrelated to your work here (@ccordoba12 and @goanpeca are working in a fix for that).
Also, seems like you did the changes using the master branch as base. If you want your changes to be released with the next version of Spyder 4.1.x you will need to rebase this PR into the 4.x branch. To do that, please follow the instructions in the contributing guide available here and rebase your work into the 4.x branch.

I have messed it up. Will be creating a new PR request .


Hi, help?

I've cloned the repo, and did git submodule update --init. But all I get is :
fatal: Not a git repository (or any of the parent directories): .git
Also, the website shows in a browser but checking what's typed doesn't work.
I am trying to set it up so it runs on my laptop once I turn off internet connection, it's for productivity reasons.
Could you help me out?

Hi, are you running git submodule update --init from the root directory of the cloned project? Once you fetch the submodule you should be able to work offline.

Thank you  for the quick response.
Just ran it here
/Documents/programmer-dvorak-master/programmer-dvorak
It seems to have worked. But then I open the index file, from webapp folder, in Chrome and it still doesn't work.

What exactly doesn't work? Can you see the index page in the browser? You should see something like this

Can you navigate to the individual lessons from there? After you update submodules you should see the output similar to this:
denisk@denisk-kubuntu:~/programmer-dvorak$ git submodule update --init
Submodule 'war/js/typingtutor' (git@github.com:denisk20/TypingTutor.git) registered for path 'src/main/webapp/js/typingtutor'
Cloning into '/home/denisk/programmer-dvorak/src/main/webapp/js/typingtutor'...
Submodule path 'src/main/webapp/js/typingtutor': checked out 'fda7eb7dee222b54d2fe07afcac3f1a0eba2a942'

After that, the lessons should work offline:

One note: if you have downloaded sources instead of cloning the repo then it won't work, because github doesn't bundle submodule projects this way. You need to clone the project by running git clone git@github.com:denisk20/programmer-dvorak.git and fetch submodules.
You could of course download the sources of TypingTutor subproject manually and copy them to src/main/webapp/js/typingtutor directory but it's much more work than cloning the repo and fetching submodules.

Problem solved. It was my fault. After updating submodules I kept opening the wrong copy of the project.
Thank you very much for creating this cool project. And thank you for your time and patience.

Glad that you solved it!
On Jan 24, 2017 12:49 AM, "Liam Beogradlija"<EMAIL_ADDRESS>wrote:

Closed #8 https://github.com/denisk20/programmer-dvorak/issues/8.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/denisk20/programmer-dvorak/issues/8#event-933650440,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC8Afim0u2Hgcjn23cfH3Win8mv9DxIuks5rVS52gaJpZM4LqgYE
.