support for PGP/MIME

Hi,
This pull request was at first dedicated to #572. I am now piling up issues from PGP/MIME milestone because it simplifies testing and integration of linked issues scattered on multiple branches. I'll start a new branch once this one will be merged on mainstream unless said otherwise.
Done

[x] fixes #571 K9Mail no longer crash when opening a S/MIME signed message ;
[x] fixes #572 Add support for PGP/MIME composing (Preference Account->Cryptography->Use PGP/MIME) ;
[x] fixes #575 Display Encrypted in message preview ;
[x] partially fixes #576 Composing of RFC-1847 message ;
[x] fixes #588 Crash when cancelling passphrase input.

What doesn't work/not tested/TODO

[ ] #576 RFC-1847 messages are shown as "Not Signed"
[ ] #575 Initialize message.encryption_type on db migration. I am not sure how messages were stored in the previous schema, so can't parse them. Any document describing the old schema?

Design notes

MessageCompose.onSend attempts a dummy encrypt/sign in order to ensure further crypto operation (on the WorkerThread) will succeed. If interactions with PGP provider are required it launches them. If no further interaction were required,  it invokes performSend which continue on a worker thread and close the window.
MessageCompose.onActivityResult fetches working Intents and invokes performSend if activity interactions were successful
MessageCompose.createMessage creates a regular message but then process the MimeMessage according to PGP/MIME specifications if cryptography was required

I'm waiting for your questions/advices. Don't hesitate.
Regards,
Alexandre

Thanks for taking the time to contribute!
This pull request is certainly large enough that it warrants having multiple commits. However, currently the commit history only reflects your work process. Please rewrite the history to group logical changes. Also, merge the commits that introduced a bug and the ones that fixed it. A clean commit history makes the code much easier to review.
Some issues I found having a first quick look:

Please follow the code style
Keep methods small
Use descriptive names for methods and fields/variables, don't abbreviate
Comments tend to drift apart from the actual implementation. Try to only add comments when you feel you need to explain why you did something, not how you did it. In most cases JavaDoc comments are unnecessary (we're not writing a library for public consumption).

Minor issues:

Ideally new functionality should be covered by (unit) tests. I know it's sometimes hard given the current state of the code base. But things like the header wrapping you introduced should definitely be covered by tests.
 Add more text as "body" if you have to.
Please start commit messages with an uppercase letter

Thank you for the feedback I'm going to fix that asap
[x] Clean git history
[x] Refactoring : small methods / method names / comments removal
[x] Unit test on MimeHeader.toWrite
[ ] May be add more tests

Do I understand correctly that you only encrypt attachments to HTML messages?
If so, that would be quite inconvenient.

I encrypt the whole mime message whatever is inside them. But I thought attachments wasn't supported on plain text message. I was actually my (bad?) interpretation of "Plain text : remove images and formating".

Just tested the use case it actually works, my bad.

@alexandre75 Great, I'm glad that we cleared that up :)

I'm done with #572 and #576
Don't hesitate if you have any questions.

Fixes #571
I also fixed #571 in this pull request because it was convenient to use the "Content-Type" parameters management methods introduced earlier.

Thanks! :)
Everything works except attach my public key.
Error message:
OpenPGP Error: Your Openpgp Provider didn't return any key on ACTION_GET_KEY, contact them.
Uncheck "Attach my public key" to proceed.

I'm using OpenKeychain 3.6.1

@Polynomdivison the implementation of "attach key" requires an enhancement of OpenKeyChain

https://github.com/open-keychain/open-keychain/commit/89d016c49be3d9258822f2803744697a0d48c80b
and
https://github.com/open-keychain/openpgp-api/commit/04b9c31dc7800307184c1561d0a3f86bfb56e11a
which is not yet officially released. Here is the build I compiled form OpenKeychain master https://www.cuenca.me/owncloud/index.php/s/jzqopUKzd1PDLgh

There is a problem when I encrypt and sign a message with k9.
Thunderbird with Enigmail correctly shows that the message is encrypted and signed by my key.
K9 shows that the message is encrypted but not signed.
When I send an email that is signed and encrypted with thunderbird and enigmail k9 correctly recognizes the message as signed and encrypted.

@Polynomdivison I could reproduce the issue, I'll come back to you when I know what's going on

I can reproduce this as well, but I use the Google Play build which doesn't support PGP/MIME yet.
I open the encrypted.asc directly in OpenKeychain and get the same problem there, so it might be on their end.
Can we narrow this down to one of the two projects?

@Polynomdivison I figured out the reason of this behavior

Enigmail/Thunderbird uses RFC-4880  to sign/encrypt (see RFC-3156 §6.2)
I chose to implement RFC-1847 (see RFC-3156 §6.1)

So, emails encrypted and signed with PGP/MIME checked will be correctly shown as signed when K9Mail will be able to display RFC-1847 mails (#576).
@clawoflight OpenKeyChain should be able to decrypt and verify the signature of the "encrypted.asc" generated by thunderbird because it is a plain OpenPGP message. Thought it won't decrypt emails encoded by this branch because RFC-1847 is out of OpenPGP scope.
Now the question is : should K9Mail implement the combined method (RFC-3156 6.2)? I thought it wasn't worth the effort because RFC1847 seemed to be the preferred one

Putting it all together
I had some work on PGP/MIME scattered among 3 branches, since peoples (me included) are testing/using my work I decided to put all the pieces together here.
This way I expect to get more feedbacks and improve the overall quality.
So I merged here the following fixes

#575 Display Encrypted in message preview
#588 Crash when cancelling passphrase input

I'm not able to send messages.
 --------- beginning of system
 --------- beginning of main
 11-16 18:40:38.173 I/k9      (15386): Running Foreground command 'getFolderUnread:xxx:INBOX', seq = 47
 11-16 18:40:38.177 I/k9      (15386): Foreground Command 'getFolderUnread:xxx:INBOX' completed
 11-16 18:40:38.255 V/k9      (15386): LockableDatabase: Transaction ended, took 5ms / com.fsck.k9.mailstore.LocalFolder.appendMessages(LocalFolder.java:1168)
 11-16 18:40:38.270 V/k9      (15386): LockableDatabase: Transaction ended, took 4ms / com.fsck.k9.mailstore.LocalFolder.appendMessages(LocalFolder.java:1168)
 11-16 18:40:38.270 E/k9      (15386): Could not save error message to K9mail-errors
 11-16 18:40:38.270 E/k9      (15386): com.fsck.k9.mail.MessagingException: Error appending message
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.mailstore.LocalFolder.saveMessage(LocalFolder.java:1291)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.mailstore.LocalFolder$15.doDbWork(LocalFolder.java:1173)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.mailstore.LocalFolder$15.doDbWork(LocalFolder.java:1168)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.mailstore.LockableDatabase.execute(LockableDatabase.java:283)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.mailstore.LocalFolder.appendMessages(LocalFolder.java:1168)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.mailstore.LocalFolder.appendMessages(LocalFolder.java:1093)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.controller.MessagingController.addErrorMessage(MessagingController.java:2668)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.controller.MessagingController.addErrorMessage(MessagingController.java:2636)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.controller.MessagingController.sendMessage(MessagingController.java:3175)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.activity.MessageCompose$SendMessageTask.doInBackground(MessageCompose.java:3347)
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.activity.MessageCompose$SendMessageTask.doInBackground(MessageCompose.java:3325)
 11-16 18:40:38.270 E/k9      (15386): 	at android.os.AsyncTask$2.call(AsyncTask.java:292)
 11-16 18:40:38.270 E/k9      (15386): 	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
 11-16 18:40:38.270 E/k9      (15386): 	at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
 11-16 18:40:38.270 E/k9      (15386): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
 11-16 18:40:38.270 E/k9      (15386): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
 11-16 18:40:38.270 E/k9      (15386): 	at java.lang.Thread.run(Thread.java:818)
 11-16 18:40:38.270 E/k9      (15386): Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String com.fsck.k9.mail.EncryptionType.name()' on a null object reference
 11-16 18:40:38.270 E/k9      (15386): 	at com.fsck.k9.mailstore.LocalFolder.saveMessage(LocalFolder.java:1265)
 11-16 18:40:38.270 E/k9      (15386): 	... 16 more

And by the other account there is an error message:
"Abort Sending" (or something like this)
--------- beginning of system
--------- beginning of main
11-16 18:43:11.315 I/k9      (15386): Running Foreground command 'getFolderUnread:xxx:INBOX', seq = 49
11-16 18:43:11.317 I/k9      (15386): Foreground Command 'getFolderUnread:xxxx:INBOX' completed
11-16 18:43:18.619 E/k9      (15386): Error sending message
11-16 18:43:18.619 E/k9      (15386): java.lang.NullPointerException: Attempt to invoke virtual method 'boolean android.widget.CheckBox.isChecked()' on a null object reference
11-16 18:43:18.619 E/k9      (15386): 	at com.fsck.k9.activity.MessageCompose.onSend(MessageCompose.java:1340)
11-16 18:43:18.619 E/k9      (15386): 	at com.fsck.k9.activity.MessageCompose.onOptionsItemSelected(MessageCompose.java:2166)
11-16 18:43:18.619 E/k9      (15386): 	at android.app.Activity.onMenuItemSelected(Activity.java:2940)
11-16 18:43:18.619 E/k9      (15386): 	at com.android.internal.policy.impl.PhoneWindow.onMenuItemSelected(PhoneWindow.java:1196)
11-16 18:43:18.619 E/k9      (15386): 	at com.android.internal.view.menu.MenuBuilder.dispatchMenuItemSelected(MenuBuilder.java:766)
11-16 18:43:18.619 E/k9      (15386): 	at com.android.internal.view.menu.MenuItemImpl.invoke(MenuItemImpl.java:152)
11-16 18:43:18.619 E/k9      (15386): 	at com.android.internal.view.menu.MenuBuilder.performItemAction(MenuBuilder.java:909)
11-16 18:43:18.619 E/k9      (15386): 	at com.android.internal.view.menu.MenuBuilder.performItemAction(MenuBuilder.java:899)
11-16 18:43:18.619 E/k9      (15386): 	at android.widget.ActionMenuView.invokeItem(ActionMenuView.java:587)
11-16 18:43:18.619 E/k9      (15386): 	at com.android.internal.view.menu.ActionMenuItemView.onClick(ActionMenuItemView.java:141)
11-16 18:43:18.619 E/k9      (15386): 	at android.view.View.performClick(View.java:4848)
11-16 18:43:18.619 E/k9      (15386): 	at android.view.View$PerformClick.run(View.java:20262)
11-16 18:43:18.619 E/k9      (15386): 	at android.os.Handler.handleCallback(Handler.java:815)
11-16 18:43:18.619 E/k9      (15386): 	at android.os.Handler.dispatchMessage(Handler.java:104)
11-16 18:43:18.619 E/k9      (15386): 	at android.os.Looper.loop(Looper.java:194)
11-16 18:43:18.619 E/k9      (15386): 	at android.app.ActivityThread.main(ActivityThread.java:5637)
11-16 18:43:18.619 E/k9      (15386): 	at java.lang.reflect.Method.invoke(Native Method)
11-16 18:43:18.619 E/k9      (15386): 	at java.lang.reflect.Method.invoke(Method.java:372)
11-16 18:43:18.619 E/k9      (15386): 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:959)
11-16 18:43:18.619 E/k9      (15386): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:754)

@Polynomdivison I fixed the bug you found.
I also found a regression introduced by #575 implementation, so I also reverted #575

There are some bugs:
When you have no key selected, pgp/mime as default ist not checked and you want to sign a message and send it:
--------- beginning of system
--------- beginning of main
11-18 17:11:35.645 E/k9      (15878): OpenPGP Error ID:0
11-18 17:11:35.645 E/k9      (15878): OpenPGP Error Message:No signing key given
11-18 17:11:35.645 E/AndroidRuntime(15878): FATAL EXCEPTION: main
11-18 17:11:35.645 E/AndroidRuntime(15878): Process: com.fsck.k9, PID: 15878
11-18 17:11:35.645 E/AndroidRuntime(15878): android.content.res.Resources$NotFoundException: String resource ID #0x7f06035e
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.content.res.Resources.getText(Resources.java:322)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.content.res.Resources.getString(Resources.java:420)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.content.res.Resources.getString(Resources.java:454)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.content.Context.getString(Context.java:395)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at com.fsck.k9.activity.MessageCompose$9.run(MessageCompose.java:1577)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.app.Activity.runOnUiThread(Activity.java:5384)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at com.fsck.k9.activity.MessageCompose.handleOpenPgpErrors(MessageCompose.java:1569)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at com.fsck.k9.activity.MessageCompose.access$1000(MessageCompose.java:133)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at com.fsck.k9.activity.MessageCompose$SignEncryptCallback.onReturn(MessageCompose.java:1561)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at org.openintents.openpgp.util.OpenPgpApi$OpenPgpAsyncTask.onPostExecute(OpenPgpApi.java:322)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at org.openintents.openpgp.util.OpenPgpApi$OpenPgpAsyncTask.onPostExecute(OpenPgpApi.java:303)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.os.AsyncTask.finish(AsyncTask.java:636)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.os.AsyncTask.access$500(AsyncTask.java:177)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.os.AsyncTask$InternalHandler.handleMessage(AsyncTask.java:653)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.os.Handler.dispatchMessage(Handler.java:111)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.os.Looper.loop(Looper.java:194)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at android.app.ActivityThread.main(ActivityThread.java:5637)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at java.lang.reflect.Method.invoke(Native Method)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at java.lang.reflect.Method.invoke(Method.java:372)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:959)
11-18 17:11:35.645 E/AndroidRuntime(15878): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:754)

When you have no key selected, pgp/mime as default ist checked and you want to sign a message and send it and Openkeychain asks to choose and key and you select none:
--------- beginning of system
--------- beginning of main
11-18 17:14:05.534 I/k9      (16148): Running Foreground command 'getFolderUnread:xxx@systemli.org:INBOX', seq = 11
11-18 17:14:05.558 I/k9      (16148): Foreground Command 'getFolderUnread:xxx@systemli.org:INBOX' completed
11-18 17:14:05.589 E/k9      (16148): Failed to create new message for send or save.
11-18 17:14:05.589 E/k9      (16148): com.fsck.k9.crypto.OpenPgpApiException: No signing key given
11-18 17:14:05.589 E/k9      (16148): 	at com.fsck.k9.message.MessageBuilder.encapsulateMimeInMultipartSigned(MessageBuilder.java:533)
11-18 17:14:05.589 E/k9      (16148): 	at com.fsck.k9.message.MessageBuilder.build(MessageBuilder.java:93)
11-18 17:14:05.589 E/k9      (16148): 	at com.fsck.k9.activity.MessageCompose.createMessage(MessageCompose.java:1201)
11-18 17:14:05.589 E/k9      (16148): 	at com.fsck.k9.activity.MessageCompose.access$2700(MessageCompose.java:133)
11-18 17:14:05.589 E/k9      (16148): 	at com.fsck.k9.activity.MessageCompose$SendMessageTask.doInBackground(MessageCompose.java:3333)
11-18 17:14:05.589 E/k9      (16148): 	at com.fsck.k9.activity.MessageCompose$SendMessageTask.doInBackground(MessageCompose.java:3325)
11-18 17:14:05.589 E/k9      (16148): 	at android.os.AsyncTask$2.call(AsyncTask.java:292)
11-18 17:14:05.589 E/k9      (16148): 	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
11-18 17:14:05.589 E/k9      (16148): 	at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
11-18 17:14:05.589 E/k9      (16148): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
11-18 17:14:05.589 E/k9      (16148): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
11-18 17:14:05.589 E/k9      (16148): 	at java.lang.Thread.run(Thread.java:818)
11-18 17:14:05.593 E/AndroidRuntime(16148): FATAL EXCEPTION: AsyncTask #8
11-18 17:14:05.593 E/AndroidRuntime(16148): Process: com.fsck.k9, PID: 16148
11-18 17:14:05.593 E/AndroidRuntime(16148): java.lang.RuntimeException: An error occured while executing doInBackground()
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at android.os.AsyncTask$3.done(AsyncTask.java:304)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:355)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at java.util.concurrent.FutureTask.setException(FutureTask.java:222)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at java.util.concurrent.FutureTask.run(FutureTask.java:242)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at java.lang.Thread.run(Thread.java:818)
11-18 17:14:05.593 E/AndroidRuntime(16148): Caused by: java.lang.RuntimeException: Failed to create a new message for send or save.
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at com.fsck.k9.activity.MessageCompose$SendMessageTask.doInBackground(MessageCompose.java:3336)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at com.fsck.k9.activity.MessageCompose$SendMessageTask.doInBackground(MessageCompose.java:3325)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at android.os.AsyncTask$2.call(AsyncTask.java:292)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	... 4 more
11-18 17:14:05.593 E/AndroidRuntime(16148): Caused by: com.fsck.k9.crypto.OpenPgpApiException: No signing key given
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at com.fsck.k9.message.MessageBuilder.encapsulateMimeInMultipartSigned(MessageBuilder.java:533)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at com.fsck.k9.message.MessageBuilder.build(MessageBuilder.java:93)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at com.fsck.k9.activity.MessageCompose.createMessage(MessageCompose.java:1201)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at com.fsck.k9.activity.MessageCompose.access$2700(MessageCompose.java:133)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	at com.fsck.k9.activity.MessageCompose$SendMessageTask.doInBackground(MessageCompose.java:3333)
11-18 17:14:05.593 E/AndroidRuntime(16148): 	... 7 more

And letters like ä ü ö are not supported.

@Polynomdivison

When you have no key selected, pgp/mime as default ist not checked and you want to sign a message and send it

The error you get when unchecking PGP/MIME wasn't introduced by this branch so better create a new issue

When you have no key selected, pgp/mime as default ist checked and you want to sign a message and send it and Openkeychain asks to choose and key and you select none

I fixed the crash when selecting <none> as signing key and use PGP/MIME is checked.

And letters like ä ü ö are not supported.

I could send sign/encrypted/plain/html emails with these letters. Can you tell me exactly how to reproduce this issue?

I could send sign/encrypted/plain/html emails with these letters. Can you tell me exactly how to reproduce this issue?

Oh when I encrypt messages it works perfectly but without encryption it looks like this:

@Polynomdivison I could reproduce the "special character" issue you found, but the bug was already there when I created this branch. I investigated further : the issue is known (#604) and fixed on master.
Thanks a lot for all these feedbacks, don't hesitate if you find something else.

@Valodim

I used an INT at first but couldn't find a trivial mapping from INT to java enum. How would you map it?
Which exceptions would you enumerate? I hesitated to catch Throwable here but finally thought Exceptions had to be caught while Error should be raised. Which throwables would you catch here?
signIntent is accessed by the UI thread and worker threads. I doubt using a cached variable would be an issue here but better be safe than sorry.

there is EnumType.ordinal()[num] to map from a number to the enum. this is considered slightly unclean sometimes, because the whole point of enums is that they should not be addressed by their ordinals, and you restrict possible changes to the enum type if you "serialize" like this, for obvious reasons. it's a code style thing - personally, I'm fine with that.
uh, all which you actually want to handle. the thing is, if you catch things like IndexOutOfBoundsExceptions and NullPointerExceptions, the app crashes less, but you also never get the crash reports so it makes the actual underlying bugs near impossible to find
maybe I'm missing it, where is it accessed from a worker thread?

Enum.values()[] that is. In my opinion, both mapping have their flows but I'll map to INT if it is the prefered way for K9Mail
My bad, I mixed up things. There is another place in the code where I want to nest RuntimeException to avoid a crash but not here. The intent here is obviously to nest application exception in RuntimeException

} catch (Exception me) {
    Log.e(K9.LOG_TAG, "Failed to create new message for send or save.", me);
    throw new RuntimeException("Failed to create a new message for send or save.", me);
}

So, I changed the code according to your proposition as I don't want to nest RuntimeException inside another RuntimeException, hence complicating the stack trace.

Worker thread : MessageCompose.SendMessageTask

@artbristol agreed, PGP_INLINE is clearer; Not sure there is a big deal is keeping the attribute final but I changed it back to final.
@Diagoras You're right, I actually stuggled to figure out which algo was used. But couldn't figure out how to specify an algo to ACTION_DETACH_SIGNATURE. Any idea how I can fix that?

@alexandre75 I think the encryption, signing, and compression algorithms are determined by OpenKeychain itself instead of by K-9 through the API. As I see it, there's only two ways that the "micalg" parameter can be populated correctly - either by scraping the hash algorithm from the detached signature, or by having OpenKeychain provide an additional return value with the ACTION_DETACH_SIGNATURE intent.

@alexandre75 Upon further investigation, it looks like the PGP block returned by the detached sign operation doesn't have the hash algorithm declared in the opening line (unlike the cleartext sign operation). Unless OpenKeychain implements a way for you to extract the signing algorithm, you'll likely just have to use SHA-512 as a default and hope for the best. :(

I finally found some time to look at this pull request. It contains a lot of stuff, including some things I want to do differently. Also, @Valodim based his pull request on this branch, but rewrote large parts of the message composition code. I'm trying to cherry-pick things from this pull request and then rebase Valodim's branch. Let's see how that goes :)
As for doing things differently; I rewrote the code to create the message preview in PR #959. Instead of saving the type of encryption in the database I chose to save the type of preview (currently: none, text, encrypted).
I'd like to get better separation of concerns. That means not putting encryption-related code inside of MimeMessage. Generally, I'd like to trim down the *Message classes instead of adding even more functionality to them.
We also have many places in the code where we do encoding/parsing of headers. Mostly, this functionality is already present in MIME4J and better than our code. So I'd rather use that than your custom Content-Type header parsing in MimeHeader.
The code in this PR still contains quite a lot of code style issues. Mostly missing spaces. I'm guessing most of them can be avoided by importing our code style settings to Android Studio as described here. You probably also want to change the template that's used when creating new files because we don't use the "Created by" comments.
I hope this doesn't sound ungrateful or discouraging. I have been working with this code base for a long time and things have been getting worse by not paying attention to details. So now I'm trying very hard to make things better and not merge every pull request that just gets the job done. Ideally every pull request leaves the code in a better state than it was in before.
We appreciate the help. Please keep the pull requests coming. @Valodim is also working on PGP/MIME, so you might want to coordinate (on IRC).

Sorry if this is a silly question, but will these updates include automatic signing and/or encryption for OpenKeychain? On the current build this is disabled.

yes.

this PR is at this point incorporated or superseded by #924, #959, #962. thanks a lot for tackling the heavy lifting required for a first iteration!


Make Swipe stop?

hey there!
would it be possible to define a swipe max? Like you can only swipe away half of the row? In addition, to keep the background and the row visible until I swipe back?
See Attachment :)

This is not as easy as it sounds.
It doesn't play very nice with the fact that you can have 2 swipe actions per direction.
It also has quite some impact on the backend code. I'm not sure if I can easily fit that in.
That's not to say I'll never implement it, but I think you'd better look for a library that already implements this if you want it on the short term.

Thank you very much. FYI: I came across this library, which exactly fitted on my needs https://github.com/baoyongzhang/SwipeMenuListView

Great stuff!
I'm happy you got your problem solved :-)


RD integration

Due to recent changes to RD API where the endpoint to check for cached info hashes was removed, there was a temporary loss of results due to this endpoint being removed. Integration was adapted to the changes, but this now basically means that Torrentio is not able to know which entries are available for instant streaming, so all links will be marked as [RD download]. That means that for anyone wanting to continue using RD integration it's best to not check the Don't show download to debrid links option.
As a naive approach to still have some indication of what's instantly streamable, Torrentio will maintain it's own short term cache of entries which were added to RD through Torrentio and were fully downloaded. This is in no means up to date with RD and might be incorrect in some cases, as RD clean up the cached files periodically, but it still should be a good enough indicator given that this status is cached temporary (8hrs). This is a temp approach that might be removed if it does not prove legit or adds too much strain on the addon itself. This will not be provided as an external api.

Can we know if this is a permanent or durable solution?

This is a temp approach that might be removed if it does not prove legit or adds too much strain on the addon itself. This will not be provided as an external api.

Not sure if something changed but I see [RD+] links just fine as of 4:38PM EST!

Not sure if something changed but I see [RD+] links just fine as of 4:38PM EST!

I've noticed that some have RD+ but other that I know had RD+ now says [RD download]
An example would be Calls (2024) show. They all say [RD DOWNLOAD] or an anime Call of the Night. They all say [RD DOWNLOAD] as well when it was [RD+]

Wouldn't a longer time duration be more appropriate, considering that RD's cache time isn't generally that short? While they probably operate something similar to an lru cache with no fixed time, I'm pretty sure its at least a day or two.

If this is the state of the api, then it does not make sense to present the RD+ links first, right? In my case I do not see anymore the files in my language, because the english ones that result as RD+ appear first in the list.

Actually, for many entries, the files in my language (Italian) are completely missing. Not even in [RD Download]

If there was a service that has API endpoints through which you can:

add magnet link to RD
check if a magnet link is cached in RD

Would you be willing to integrate that into Torrentio? It would benefit the whole community.

This post mentions RealDebrid (I'm assuming RD stands for RealDebrid?) but I am seeing similar behavior with AllDebrid. All links are maked with "[AD download]" despite many of the links being cached. Is this the same issue or is it unrelated?


Run challenge on different local port

Hello,
applications like Skype etc. often reserve the port 80.
The challenge runs over an nginx-gateway (hostname based -> nginx as proxy) so i can send the challenge to any other local port.
So my question is, is it possible to use "officially" port 80 (LE-Server will see port 80) but tell the client that it should listen on port xyz?
Thanks

This was added in version 1.9.9!


Multiple languages for labels

ODK labels support multiple languages, as do Stata labels; yet odkmeta imports only one language. Add multiple-language functionality for variable and value labels.

I haven't seen any requests for this, so I'm going to go ahead and close this issue.


Allow control "Double tap to zoom" & "Pinch to zoom" separately

Duplicate of #32 ?

Tracking this in https://github.com/saket/telephoto/issues/32


A little help to make this shader in ShaderGraph?

I tried to port it to UE4 material. But failed for unknown reason.
Then I am thinking if it can work in the similar node editor: ShaderGraph in Unity.
But still not luck.
It looks like the image below:
https://i.imgur.com/IvCBVe1.png
Would you mind to help me figure this out?
Using 2021.1.9f1 and Universal rendering pipeline.

I'm not familiar with shader graph, but I might be able to help. It looks like the first two "Transform" nodes in your graph are transforming from World space to View space. In my shader code, I transform from Object space to View space. This might be the problem.

I'm not familiar with shader graph, but I might be able to help. It looks like the first two "Transform" nodes in your graph are transforming from World space to View space. In my shader code, I transform from Object space to View space. This might be the problem.


Fix/not applicable color

FIx LTS emissions card colors:
Extra: Fix shared modal on overview. Now is copying the right URL (f.e: http://localhost:3000/embed/ndc-overview/1)

Share button on embed overview page doesn't work:

Thanks!


mod_webauthn letting sessions expire while in use

I have a test server using database providers with the following apache config elements:
WebauthnLoginPath /authn/preauth
WebauthnSessionPath /authn/session
WebauthnCookieName webauthn
WebauthnMaxCacheSeconds 60
...
<Location "/authn" >
   Satisfy Any
   Allow from all
   WSGIProcessGroup webauthn
</Location>
...
<Location "/ermrest" >
  AuthType webauthn
  Require valid-user
   WSGIProcessGroup ermrestddl
</Location>

<LocationMatch "/ermrest/authn">
  AuthType None
  Require all granted
</LocationMatch>

<LocationMatch "/ermrest/catalog/[^/]+/(entity|attribute|attributegroup|aggregate)" >
 AuthType webauthn
  Require valid-user
   WSGIProcessGroup ermrest
</LocationMatch>

I run tests with a procedure like:

Login using curl -c cookie -d username=testuser -d password=dummypassword https://server.example.com/authn/session
Run test scripts using curl -b cookie https://server.example.com/ermrest/...

The webauthn server is configured with a 30 minute session expiration time.
If I repeat step 2 above a number of times with some delays, I've observed situations where the launch of a test script will successfully perform dozens of operations with a valid session and then suddenly report an unauthorized error and stop.  These are back-to-back tests with millisecond-level delay between requests that succeeded and the ones that failed after.
I have to repeat step 1 to login again to proceed.
If I check the server ssl_access_log, I don't think I see any POST requests to the /authn/session resource other than the ones I performed with curl.  It seems that mod_webauthn is never telling the resource to extend its lifetime.

I just wanted to note that I am still seeing this behavior as of latest webauthn/master at 4405082ec547a5395d55ad86c94264c6446e8c55

I've re-tested and confirmed that this is now working.


Feature Request (Submitted by 3723)

Description
Binder adjustment: Display amount of empty card slots in a binder in a better way.
Desired Functionality
I would find it more intuitive to not see 8 pages of empty cards when a binder only has e.g. 4-5 cards in it. It's not really necessary and somewhat confusing this way. Also when adding new cards or when adding your first card to a new binder, instead have something like "add a card" instead of having all 8 pages of cards that you have to click on individually. I hope that's understandable, if not ask me and I try to explain it better.

with how its designed it might be tough to implement. As we preferred allowing user's the flexibility to place card in any of the 75 spots, we prepopulate all 75 spots to give them that flexability. Will have to think about it


Make tester work with @types feature

There upcoming feature in the upcoming Typesscript 2.0 which allow use node_modules/@types folder to store TypeScript definitions and there already exists porting efforts to that stratecy of deliverying TypeScript definitions. See https://github.com/DefinitelyTyped/DefinitelyTyped/tree/types-2.0
Current tester not working with new definitions because it could not resolve definitions using ///<reference types="..." /> format.
Would be reasonable to make this work for this scenario too.
I make this work by modifying src/tsc/Tsc.ts file by including additional command line options --typeRoots . --types jquery. That's very dirty, and just to give proof of concept. Full fix should include additional parameter which to dt and some other changes probably should be required on TypeScript side.
Now more about the change:

Make root folder of DefintelyTyped as source folder for definitions --typeRoots .
Fix error in the latest TypeScript which gives following error

error TS2688: Cannot find type definition file for '.git'.
error TS2688: Cannot find type definition file for 'gsap'.
error TS2688: Cannot find type definition file for 'node_modules'.
error TS2688: Cannot find type definition file for 'scripts'.
error TS2688: Cannot find type definition file for 'swipe.js'.

if some dummy existing types module does not provided. That's why --types jquery

@RyanCavanaugh ?

Already in progress


Windows: Error when editing HTML files in Atom

Error: 'node' is not recognized as an internal or external command, operable program or batch file.
Error: 'node' is not recognized as an internal or external command,
operable program or batch file.
at ChildProcess. (C:\Users\gemma.atom\packages\linter-bootlint\node_modules\sb-exec\lib\index.js:56:20)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:191:7)
at maybeClose (internal/child_process.js:877:16)
at Socket. (internal/child_process.js:334:11)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at Pipe._handle.close [as _onclose] (net.js:493:12)

It looks like the issue being reported is from the linter-bootlint package rather than the lipsum package (it's a known issue with that package - https://github.com/AtomLinter/linter-bootlint/issues/92 - you will need to install node.js locally for that package to work without throwing this error).
If this is being reported by your system as an issue with the lipsum package could you try disabling linter-bootlint and let me know if the issue is still being reported?

Oops! My bad... I must have been tired and distracted cos it looks like I posted this for the wrong package. My apologies. :/ But thanks for the heads-up. I'll install node.js and see if that resolves the issue. :)


In-app purchase <IP_ADDRESS>

The price was originally 3USD/year but for now, I have added an introductory price so that it is only 1.49USD for the first year for everyone because the new version is not yet done. It is my first time doing subscriptions so I need to do more testings.

[x] Implement IAP in but sell it at a lower cost (50% for a whole year while I work on the new version), not yet working for iOS
[x] Pass IAP test on IOS and Android, not yet working on IOS
[x] Lock RS and More Info depending on whether it is pro version, also lock main account feature as well
[x] Keep checking IAP status and adjust pro version everyday
[x] Validate purchase receipt maybe later
[x] Auto pop if the purchase was successful

All good.


[Feature Request] (New) Sentora Module

https://github.com/boxbilling/boxbilling/issues/1091

REally ? Sentora is a discontinued project

REally ? Sentora is a discontinued project

Is it entirely though?
It's definitely not moving very fast and the current release is probably not something that anyone should use in production, but there does seem to be some work going on somewhere in the background, and there was a public beta release of a v2 at the start of this year. See http://forums.sentora.org/showthread.php?tid=12794
I don't know much more about it than that, but it might not be completely dead, possibly, maybe.

Closed and added to #1362


Specialize parser for Iter=&[u8]

This improves parser performance by 30-60% (that's right, more than 2x faster for many inputs). Performance when reading from other types of iterators is not affected.
The tests are going to require a Serde release containing https://github.com/serde-rs/serde/pull/422 and https://github.com/serde-rs/serde/pull/423.

There are some contortions to keep the new Read trait out of the public API and to not affect the type parameters on the Deserializer struct. This lets us replace the manual specialization with actual specialization once that feature is stable.

Those improvements are impressive. Code looks good to me.

Coverage remained the same at 48.511% when pulling 5dcda93c2822bffc7b82fb7758a88a1902589021 on dtolnay:sliceread into 12f617d966e2169bfe90bb26397bc1ea7c124501 on serde-rs:master.


reduce_max on empty int32 array returns -2147483648

System information

Have I written custom code: no
OS Platform and Distribution: Linux
TensorFlow installed from (source or binary): source
TensorFlow version: 1.14.0
Python version: 2.7

Describe the current behavior
# Call reduce_max on an empty Tensor.
x = tf.reduce_max(tf.zeros([0], tf.int32))
with tf.Session() as sess:
  print x.eval()
# prints -2147483648

Describe the expected behavior
Would be better if reduce_max threw an error when fed an empty Tensor.

I have tried on colab with TF version 1.14 and nightly versions was able to reproduce the issue.Please, find the gist here.Thanks!

The test run into this snippet of code
https://github.com/tensorflow/tensorflow/blob/0b180e77223a1819758945a9d09c6960c553a1f3/tensorflow/core/kernels/reduction_ops_common.h#L189-L194

@jvishnuvardhan
I believe we can close this issue now. Since the behavior is as expected.

I am closing this issue as it is intended behavior. Thanks!


Sonar Credentials

How does the sonar collector connect to our sonarqube server without credentials? When do you setup the username and password for sonar?

We do not support (yet) auth based collector for sonar.
Will take this as an enhancement.

Created PR to add sonar basic auth support but so far no feedback #512

we will review it tomorrow, thanks for your cooperation

Closed per comment by subesokun.


BAU Allow webhooks logs to go through to Splunk

The lambda responsible for ingesting output from fargate instances
requires an environment attribute to be set (using commons structured
logging). This sets the environment and container attributes for all
messages logged by the service which will allow these lines to be
ingested.

@alexbishop1 thanks for the review, moved the commons dependency and kept the dropwizard where it is (no explicit versions needed)


SSR

Morning, does this work with ssr on the server or with static sites.

Hi, sorry for the late reply. It works on both, the ssr & client mode. Feel free to try and test the nuxt-lazysizes module 😉👍


Notification when new update version is avaible

Is it possible to add a notification when you open the app and new update version is avaible with redirect link to releases?

It should update automatically on launch if you let it open for a minute.

Oh! I didn't knew that.
Then maybe a notification: new version avaible and updating or something like that. It will be kinda useful to know when the app is going to update, in what version..etc, even better with changelog.
If you find it's difficult or uncessery to be implemented no worries.
Thanks for your clarification anyway.👍

A notification makes sense, I'll implement it. Thanks.


Improve UI/UX of Send max option

When using the Send max option, the UI/UX can be improved.

Version MD5:<PHONE_NUMBER>8911AD8CF97D2867499956
Device: Moto G5
Screen size: 5 inch
Pixels: 1080 x 1920
Ratio: 16:9
OS: Android 7.0

The solution also has to look good in case you are using the USD converter:

Suggestion: stack the text on top of each other
instead of:
SEND MAX
it could be:
SEND
MAX

The new version is a big improvement, but still some UI/UX improvements possible. As you can see the DCR input turns into a scrolling field, which only displays half of the numbers.

Since it's about people's money, I think we should strive to show the full number (rather than cut it in half and make the user scroll to check whether it is the right amount).

Here is a screenshot of the new input field on my device.
Wallet version: 18E7C5BBA2B6F62C3CF093B97059EC87

USD amount is scrolling down, while DCR amount is scrolling sideways


Tweak layout fixes

Some of the recent merge has introduced a layout issue in the Browse Page.
This PR fix that problem.

thanks!


Unable to get running

Describe the bug
Doing a clean install. Followed the getting started guide. The login UI gets stuck on Admin node loading... API and Core node have a checkmark. Eventually it times out and takes me to the login page. I create a user but it throws an "invalid credentials" error. I create another user and that one works.
To Reproduce
Steps to reproduce the behavior:

Download release zip
Unzip
Run docker compose up -d
See error

Expected behavior
Expect it to start up properly and take me to the login page
Screenshots

Desktop (please complete the following information):

OS: Ubuntu Server 23.10
Browser: Chrome
Version: 1.2.0

Logs
Run those commands and attach result to the ticket:
docker_ps.txt
docker_compose_logs.txt
Additional context
The log is repeating 2024-05-09 15:32:25.272  INFO 7 --- [nio-8080-exec-1] o.s.s.o.p.token.store.JdbcTokenStore     : Failed to find access token I've tried searching for that error but haven't found anything

It just started working after several reboots


Error when running grounded_sam_simple_demo.py

python3 grounded_sam_simple_demo.py
2023-06-05 12:22:36.631833: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-05 12:22:37.323961: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/olivia/Segment-Everything-Everywhere-All-At-Once/demo_code/seem/lib/python3.9/site-packages/cv2/../../lib64:/usr/local/cuda-11.6/lib64:/usr/local/cuda-11.6/lib:
2023-06-05 12:22:37.324024: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/olivia/Segment-Everything-Everywhere-All-At-Once/demo_code/seem/lib/python3.9/site-packages/cv2/../../lib64:/usr/local/cuda-11.6/lib64:/usr/local/cuda-11.6/lib:
2023-06-05 12:22:37.324032: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/home/olivia/Segment-Everything-Everywhere-All-At-Once/demo_code/seem/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
final text_encoder_type: bert-base-uncased
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias']

This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
╭───────────────────────── Traceback (most recent call last) ──────────────────────────╮
│ /home/olivia/Segment-Everything-Everywhere-All-At-Once/demo_code/Grounded-Segment-An │
│ ything/grounded_sam_simple_demo.py:25 in                                     │
│                                                                                │
│    22 grounding_dino_model = Model(model_config_path=GROUNDING_DINO_CONFIG_PATH, mod │
│    23                                                                                │
│    24 # Building SAM Model and SAM Predictor                                         │
│ ❱  25 sam = sam_model_registrySAM_ENCODER_VERSION  │
│    26 sam_predictor = SamPredictor(sam)                                              │
│    27                                                                                │
│    28                                                                                │
│                                                                                │
│ /home/olivia/Segment-Everything-Everywhere-All-At-Once/demo_code/Grounded-Segment-An │
│ ything/segment_anything/segment_anything/build_sam.py:15 in build_sam_vit_h          │
│                                                                                │
│    12                                                                                │
│    13                                                                                │
│    14 def build_sam_vit_h(checkpoint=None):                                          │
│ ❱  15 │   return _build_sam(                                                         │
│    16 │   │   encoder_embed_dim=1280,                                                │
│    17 │   │   encoder_depth=32,                                                      │
│    18 │   │   encoder_num_heads=16,                                                  │
│                                                                                │
│ /home/olivia/Segment-Everything-Everywhere-All-At-Once/demo_code/Grounded-Segment-An │
│ ything/segment_anything/segment_anything/build_sam.py:105 in _build_sam              │
│                                                                                │
│   102 │   sam.eval()                                                                 │
│   103 │   if checkpoint is not None:                                                 │
│   104 │   │   with open(checkpoint, "rb") as f:                                      │
│ ❱ 105 │   │   │   state_dict = torch.load(f)                                         │
│   106 │   │   sam.load_state_dict(state_dict)                                        │
│   107 │   return sam                                                                 │
│   108                                                                                │
│                                                                                │
│ /home/olivia/Segment-Everything-Everywhere-All-At-Once/demo_code/seem/lib/python3.9/ │
│ site-packages/torch/serialization.py:797 in load                                     │
│                                                                                │
│    794 │   │   │   # If we want to actually tail call to torch.jit.load, we need to  │
│    795 │   │   │   # reset back to the original position.                            │
│    796 │   │   │   orig_position = opened_file.tell()                                │
│ ❱  797 │   │   │   with _open_zipfile_reader(opened_file) as opened_zipfile:         │
│    798 │   │   │   │   if _is_torchscript_zip(opened_zipfile):                       │
│    799 │   │   │   │   │   warnings.warn("'torch.load' received a zip file that look │
│    800 │   │   │   │   │   │   │   │     " dispatching to 'torch.jit.load' (call 'to │
│                                                                                │
│ /home/olivia/Segment-Everything-Everywhere-All-At-Once/demo_code/seem/lib/python3.9/ │
│ site-packages/torch/serialization.py:283 in init                                 │
│                                                                                │
│    280                                                                               │
│    281 class _open_zipfile_reader(_opener):                                          │
│    282 │   def init(self, name_or_buffer) -> None:                               │
│ ❱  283 │   │   super().init(torch._C.PyTorchFileReader(name_or_buffer))          │
│    284                                                                               │
│    285                                                                               │
│    286 class _open_zipfile_writer_file(_opener):                                     │
╰──────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central
directory

I would appreciate any assistance with this error.

Fixed directory paths


Feat(button): Added an option to start playing again when airplay tar…

With this option player will automatically start playing when the user has selected a new airplay device.

Hello @jontore,
I believe autoplay already happens by default when changing AirPlay targets. Is there a specific case where autoplay does not happen that this change is meant to address, or some other reason for the PR?
If so, could you please provide a minimal, complete, verifiable example (MCVE) that demonstrates the issue? That will help us to review this PR. Thanks!


BP-2349: Base layout

Base for the new layout (new structure and styles but without cleanup)
Old design should not be impacted
new design can be accessed by adding ?redesign to the url

Did a little cleanup but more will follow in other tickets as I did it there already

Fixed the imports so building works


Add worker

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #44 into r2.0 will decrease coverage by 1.37%.
The diff coverage is 82.85%.

@@            Coverage Diff            @@
##            r2.0      #44      +/-   ##
=========================================
- Coverage   96.5%   95.12%   -1.38%
=========================================
  Files          7       11       +4
  Lines        315      349      +34
  Branches      22       22
=========================================
+ Hits         304      332      +28
- Misses         7       13       +6
  Partials       4        4

Impacted Files
Coverage Δ

src/sagemaker_containers/transformer.py
0% <0%> (ø)

src/sagemaker_containers/status_codes.py
100% <100%> (ø)

src/sagemaker_containers/worker.py
100% <100%> (ø)

src/sagemaker_containers/__init__.py
100% <100%> (ø)
:arrow_up:

src/sagemaker_containers/content_types.py
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d3f3e6...608daa6. Read the comment docs.


Android-25, View$InvalidateInfo not found

I have put the android-25 android.jar in my platforms/android-25 folder, we have some projects that work but some of them get this error when building which I'm not sure how to proceed on.
error: cannot access InvalidateInfo
class file for android.view.View$InvalidateInfo not found
Consult the following stack trace for details.
com.sun.tools.javac.code.Symbol$CompletionFailure: class file for android.view.View$InvalidateInfo not found

same here. And data binding doesn't work


Improve gRPC context propagation with hpack

The earlier support for Go gRPC context propagation didn't hpack encode the traceparent header and value, which meant we needed 69 (55+11+3) bytes to encode the traceparent information. 55 for the actual traceparent value, 11 for the "traceparent" string and 3 for flags and sizes. By default the gRPC internal buffer starts at 64 and grows to 128, so we never had enough space in the request buffers to add our 69 bytes on the first call for a connection. This was problematic for gRPC calls that don't reuse connections.
To make this better I added hpack encoding of the values, which conveniently fits within the small gRPC buffers. The only downside is that I had to use bpf_loop and restrict this support to kernels >= 5.17. We may find a better way in the future, but this is a limitation for now. After browsing various OSS projects, closing of the gRPC connection after each request is common.

I need to clean-up the loop code a bit, I realized I forgot to fix some of the hardcoded 55s which were added because the verifier was unhappy.

Codecov Report
Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (77e0708) 79.36% compared to head (b5c7713) 45.13%.

Files
Patch %
Lines

pkg/internal/ebpf/common/common.go
0.00%
5 Missing :warning:

pkg/internal/ebpf/grpc/grpc.go
0.00%
5 Missing :warning:

pkg/internal/ebpf/httpfltr/httpfltr.go
0.00%
2 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #531       +/-   ##
===========================================
- Coverage   79.36%   45.13%   -34.23%
===========================================
  Files          69       67        -2
  Lines        5772     5603      -169
===========================================
- Hits         4581     2529     -2052
- Misses        970     2921     +1951
+ Partials      221      153       -68

Flag
Coverage Δ

integration-test
?

unittests
45.13% <0.00%> (-0.11%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Amazing!! Even if we need Kernel 5.17+ for hpack, will users with lower kernel versions still be able to use grpc tracing with limited functionality?

Yes, basic gRPC tracing will work, just not context propagation :(. Maybe we figure out a better approach in the future.


Errors on documentation

I couldn't find anywhere to create a merge request for the documentation.
https://vault.koudingspawn.de/install-vault-crd
base64 -D should be base64 -d
KUBERNETES_VAULT_PATH is not described in the documentation but implemented in /src/main/java/de/koudingspawn/vault/vault/VaultConfiguration.java:62.
A simple description could be:
Please specify here the Vault auth path to be used. Defaults to kubernetes.

Hi @Sayrus,
thanks a lot for your feedback. I've updated the documentation, it is a hosted gitbook at gitbook.com, so I don't know if it is possible to send merge requests there.


AE-character kills the cow

Hi,
A colon-separated csv contains this value in a given row:
 mr. Æ.baron van Mackay

Uploading the CSV in COW results in an error.
Removing the row containing the special character fixes the problem
Opening the file in OpenOffice and saving as UTF-8 fixes the problem

Is it possible to make the last step automagical? Or someother way to circumvent this issue?
Best,
Richard

COW tells you what encoding it thinks the file is, right? What encoding is it and does it match the actual encoding of the file?
If the answer to the final question is "no", this might be difficult to fix. Something to consider is giving csvw_tool an "--encoding" argument that would force the correct encoding.
Final thought: does cow work in python3? I've fixed encoding nightmares before by switching pythons

You are right, cow does sniff out the encoding and the new version of cow (as released to pipy this morning) uses this encoding to correctly handle the file.
In python3 this would have been a bit easier, but currently cow is still written in python2.7, so it required an additional import.
The new version of Cow is also being used by cattle as of today, so the Æ character (or any other special character) should not be a problem anymore.


Improve case sensitivity section for SQLite provider

This section's wording is a bit confusing on whether or not SQLite supports case-insensitive filtering or not.
It should be clear that the default filtering mode is case-sensitive and that SQLite supports case insensitive only if your data is pure ASCII characters.
It's also not clear how I would actually use this workaround:
CREATE TABLE mytable (
    sample TEXT COLLATE NOCASE   /* collating sequence NOCASE */
);

So might be a good idea to include a resource that clarifies where to include this snippet

Found this as I have no idea how to use COLLATE NOCASE


how to use the generated results for MAYA

After runing the demo code, I can get the .obj and .npz files.
How to put these files in MAYA for further usage.
Thanks

Please refer to the SMPL for Maya in https://smpl.is.tue.mpg.de/ (MPI copy right) for using SMPL parameter results in MAYA.

Thanks for your relpy.
Sorry Im new for SMPL, it seems it needs .fbx file for MAYA. how to change .obj file to .fbx file

Thanks for your relpy.
Sorry Im new for SMPL, it seems it needs .fbx file for MAYA. how to change .obj file to .fbx file

you can do that in Maya

@CrossLee1 @yahiakamal
I have uploaded a new version which support the FBX animation please refer to export-to-blender-fbx.


request.get_data() is always empty because of flask/werkzeug bug with chunked transfer encoding

The python3-flask template (and likely the python2) is bitten by this Flask issue:  https://github.com/pallets/flask/issues/2229
Flask / Werkzeug have a bug handling chunked transfer encoding--and perhaps any other time that Content-Length is not set--when combined with WSGI.
When the request is forwarded through the of-watchdog process, http.NewRequest is given a reference to the body io.ReadCloser.  Since that is a stream, the outbound request will always use chunked transfer encoding and no Content-Length.
This could be avoided by changing of-watchdog to read the whole body into a buffer and then send it unchunked; but it would make more sense, I think, either to upgrade to a fixed version of Flask/Werkzeug (assuming there is one) or revert back to Flask's development server (app.run(...)) rather than using gevent's WSGIServer, as the built-in server does not seem to be affected.
The following examples are using the same of-watchdog and function code.  The only difference is that the first one is using Flask with WSGIServer and the second is using the Flask built-in server.
WSGI server
$ curl -i -XPOST -d "rawdsddddddddddddddddddddddd" 'http://<IP_ADDRESS>:9999/'
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Wed, 25 Jul 2018 23:33:36 GMT

Flask's server
$ curl -i -XPOST -d "rawdsddddddddddddddddddddddd" 'http://<IP_ADDRESS>:9999/'
HTTP/1.1 200 OK
Content-Length: 28
Content-Type: text/html; charset=utf-8
Date: Wed, 25 Jul 2018 23:31:29 GMT
Server: Werkzeug/0.14.1 Python/3.6.5

rawdsddddddddddddddddddddddd

I recently started to try and build some new functions and use this template and I've run into this same problem.

I was able to get this working by applying a suggestion from https://github.com/benoitc/gunicorn/issues/1733#issuecomment-377000612
@app.before_request
def fix_transfer_encoding():
    app.logger.debug('Headers: %s', request.headers)
    app.logger.debug('Body: %s', request.get_data())
    """
    Sets the "wsgi.input_terminated" environment flag, thus enabling
    Werkzeug to pass chunked requests as streams.  The gunicorn server
    should set this, but it's not yet been implemented.
    """

    transfer_encoding = request.headers.get("Transfer-Encoding", None)
    if transfer_encoding == u"chunked":
        request.environ["wsgi.input_terminated"] = True

Plain Flask was both simpler to use and quicker to build, but is not said to be production-quality.
I'll take a look at this proposal.

@ssullivan I'm talking with @LucasRoesler about this. Do we want to revert to "plain flask" which is much faster at build time and doesn't need the work-around?

At the moment I’m not familiar with the performance differences between plain flask and the current implementation. If it is more performant and doesn’t have the problem with chunked encoding it seems to make sense to switch to that.

I think for short term, a working template is pretty important.  But in the long term it would be good to use wsgi.
Have we considered using conda instead of pip.  Using conda and the conda-forge could greatly speed up the install of packages like wsgi

I was able to get this working by applying a suggestion from benoitc/gunicorn#1733 (comment)
@app.before_request
def fix_transfer_encoding():
    """
    Sets the "wsgi.input_terminated" environment flag, thus enabling
    Werkzeug to pass chunked requests as streams.  The gunicorn server
    should set this, but it's not yet been implemented.
    """

    transfer_encoding = request.headers.get("Transfer-Encoding", None)
    if transfer_encoding == u"chunked":
        request.environ["wsgi.input_terminated"] = True

you saved me hours ! Thank you 👍

Michael is that not in the OpenFaaS templates yet or are you wanting the hack for something else?

We also have RAW_BODY available now as an environment variable which may also be useful here

Hey @alexellis , thanks for reaching out! I came here from using the DockerHub Image: tiangolo/uwsgi-nginx-flask. I had the exact same problem there and struggled a lot to find a solution. Maybe there might be a better way to solve this issue, but it works perfectly.


Google io '19 - ML Kit x Material Design Demo

https://youtu.be/bEsncYGeMF8?t=1966
at the session the speaker mentioned that demo is open sourced on the flutter repo , but it'snot ! can u upload to the shrine repo in examples in flutter/flutter
or in the samples repo ?

This has moved to https://github.com/FirebaseExtended/flutterfire/tree/master/packages/firebase_ml_vision/example


Received valid JSON from LLM, but JSON was missing fields: 'function' error help.

Describe the bug
I am getting a function error when testing the connection to Ollama. It seems I get a error staying. To me it seems the because of the model and Ollama results it's not working with Memgpt so it's creates a error.
`~$ memgpt run
[nltk_data] Downloading package punkt_tab to
[nltk_data]     /home/vivienne/.local/lib/python3.10/site-
[nltk_data]     packages/llama_index/core/_static/nltk_cache...
[nltk_data]   Unzipping tokenizers/punkt_tab.zip.
? Would you like to select an existing agent? Yes
? Select agent: TremendousSeashell
🔁 Using existing agent TremendousSeashell
Hit enter to begin (will request first MemGPT message)
An exception occurred when running agent.step():
Traceback (most recent call last):
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/local_llm/llm_chat_completion_wrappers/dolphin.py", line 231, in output_to_chat_completion_response
function_name = function_json_output["function"]
KeyError: 'function'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/local_llm/chat_completion_proxy.py", line 191, in get_chat_completion
chat_completion_result = llm_wrapper.output_to_chat_completion_response(result)
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/local_llm/llm_chat_completion_wrappers/dolphin.py", line 234, in output_to_chat_completion_response
raise LLMJSONParsingError(f"Received valid JSON from LLM, but JSON was missing fields: {str(e)}")
memgpt.errors.LLMJSONParsingError: Received valid JSON from LLM, but JSON was missing fields: 'function'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/main.py", line 459, in run_agent_loop
new_messages, user_message, skip_next_user_input = process_agent_step(user_message, no_verify)
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/main.py", line 427, in process_agent_step
new_messages, heartbeat_request, function_failed, token_warning, tokens_accumulated = memgpt_agent.step(
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/agent.py", line 800, in step
raise e
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/agent.py", line 698, in step
response = self._get_ai_reply(
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/agent.py", line 409, in _get_ai_reply
raise e
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/agent.py", line 378, in _get_ai_reply
response = create(
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/llm_api/llm_api_tools.py", line 223, in wrapper
raise e
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/llm_api/llm_api_tools.py", line 196, in wrapper
return func(*args, **kwargs)
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/llm_api/llm_api_tools.py", line 464, in create
return get_chat_completion(
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/local_llm/chat_completion_proxy.py", line 194, in get_chat_completion
raise LocalLLMError(f"Failed to parse JSON from local LLM response - error: {str(e)}")
memgpt.errors.LocalLLMError: Failed to parse JSON from local LLM response - error: Received valid JSON from LLM, but JSON was missing fields: 'function'
? Retry agent.step()? No

Enter your message: How are you?

An exception occurred when running agent.step():
Traceback (most recent call last):
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/local_llm/llm_chat_completion_wrappers/dolphin.py", line 231, in output_to_chat_completion_response
function_name = function_json_output["function"]
KeyError: 'function'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/local_llm/chat_completion_proxy.py", line 191, in get_chat_completion
chat_completion_result = llm_wrapper.output_to_chat_completion_response(result)
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/local_llm/llm_chat_completion_wrappers/dolphin.py", line 234, in output_to_chat_completion_response
raise LLMJSONParsingError(f"Received valid JSON from LLM, but JSON was missing fields: {str(e)}")
memgpt.errors.LLMJSONParsingError: Received valid JSON from LLM, but JSON was missing fields: 'function'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/main.py", line 459, in run_agent_loop
new_messages, user_message, skip_next_user_input = process_agent_step(user_message, no_verify)
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/main.py", line 427, in process_agent_step
new_messages, heartbeat_request, function_failed, token_warning, tokens_accumulated = memgpt_agent.step(
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/agent.py", line 800, in step
raise e
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/agent.py", line 698, in step
response = self._get_ai_reply(
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/agent.py", line 409, in _get_ai_reply
raise e
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/agent.py", line 378, in _get_ai_reply
response = create(
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/llm_api/llm_api_tools.py", line 223, in wrapper
raise e
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/llm_api/llm_api_tools.py", line 196, in wrapper
return func(*args, **kwargs)
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/llm_api/llm_api_tools.py", line 464, in create
return get_chat_completion(
File "/home/vivienne/.local/lib/python3.10/site-packages/memgpt/local_llm/chat_completion_proxy.py", line 194, in get_chat_completion
raise LocalLLMError(f"Failed to parse JSON from local LLM response - error: {str(e)}")
memgpt.errors.LocalLLMError: Failed to parse JSON from local LLM response - error: Received valid JSON from LLM, but JSON was missing fields: 'function'
?`
Please describe your setup

[ ] How did you install memgpt?
pip install pymemgpt

[ ] Describe your setup

What's your OS (Windows/MacOS/Linux)?
I have Ubuntu 22.04 running with 16 gigs of ram and a Nvidia gtx 1660 super. Currently Ollama and Memgpt is installed. It's not powerful but it can run AI.

How are you running memgpt? (cmd.exe/Powershell/Anaconda Shell/Terminal)

I am running from command line
Screenshots
I know memgpt can connect to the ollama ai interface with the dolphin mistral model.
Additional context
I tried both wrapper for mistral and airboros. Similiar error.
MemGPT Config
Please attach your ~/.memgpt/config file or copy past it below.
[defaults]
preset = memgpt_chat
persona = sam_pov
human = basic
[model]
model = dolphin2.2-mistral:7b-q6_K
model_endpoint = http://localhost:11434
model_endpoint_type = ollama
model_wrapper = airoboros-l2-70b-2.1
context_window = 8192
[embedding]
embedding_endpoint_type = local
embedding_model = BAAI/bge-small-en-v1.5
embedding_dim = 384
embedding_chunk_size = 300
[archival_storage]
type = chroma
path = /home/vivienne/.memgpt/chroma
[recall_storage]
type = sqlite
path = /home/vivienne/.memgpt
[metadata_storage]
type = sqlite
path = /home/vivienne/.memgpt
[version]
memgpt_version = 0.3.25
[client]
anon_clientid = 00000000-0000-0000-0000-000000000000

If you're not using OpenAI, please provide additional information on your local LLM setup:
Local LLM details
If you are trying to run MemGPT with local LLMs, please provide the following information:

[ ] The exact model you're trying to use (e.g. dolphin-2.1-mistral-7b.Q6_K.gguf)

[ ] The local LLM backend you are using (web UI? LM Studio?)

I am using Ollama. I have not figured how to add a web ui that I can access memgpt or ollama chat at the same time.

[ ] Your hardware for the local LLM backend (local computer? operating system? remote RunPod?)
                            system         Computer

/0                                bus            Motherboard
/0/0                              memory         16GiB System memory
/0/1                              processor      Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
/0/100                            bridge         2nd Generation Core Processor Family DRAM Controller
/0/100/1                          bridge         Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port
/0/100/1/0                        display        TU116 [GeForce GTX 1660 SUPER]
/0/100/1/0.1     card1            multimedia     TU116 High Definition Audio Controller

I was able to get Illama LLM to run some but I notice similar errors. It seems several jsons have functions and params sent to MemGPT. It is causing multiple problems. I tried llama.cpp and Ollama with MemGPT. I prefer MemGPT over ChatGPT but is there is a LLM that work with llama.cpp + MemGPT

These are some of the common problems
"Failed to decode valid MemGPT JSON from LLM output:\n=====\n{raw_llm_output}\n=====")
"Failed to parse JSON from local LLM response - error: {str(e)}")
Failed to decode JSON from LLM output:   "function":  - error
``

These are the common ones.


Import v1alpha2 instead of v1alpha1

/kind feature
Describe the solution you'd like
We need to import the new cluster-api types and not import the old cluster-api types.
/assign
/lifecycle active
/priority important-soon

/milestone support v1alpha2


order of enableJsonFlag in code affects whether or not flags are registered

Describe the bug
Specifying enableJsonFlag after flags, causes the flags to not register. This worked fine in the last version I tried, 1.9.5, but has changed in 1.12.0.
To Reproduce
Steps to reproduce the behavior:
will only register the json flag
static flags = {
  test: Flags.string()
}
static enableJsonFlag = true

will register both flags
static enableJsonFlag = true
static flags = {
  test: Flags.string()
}

@vidarc Thanks for finding this issue. I've got a PR open now to fix it #450


"CASMonitoring" should change to "SSO Sessions Report"

Also when I browse over the title, it grays out and becomes invisible, sort of.

Completed - local repo


Q: Comparison with kd-tree?

How does performance compare to a kd-tree?
In particular, I am wondering if this data structure could be used for example for photon mapping.

Hello,
I am not sure if TreeNSearch can even be used for photon mapping since, as far as I understand it, you would need to do queries with rays and we do not support rays. TreeNSearch exclusively works with point clouds and its goal is to find neighbors between the points.

I don't want to make any absolute claims about the efficiency of different data structures for different applications. In our very specific case, the neighborhood search stage in SPH simulations with millions of particles, using an octree was faster than a kdtree. But we had to implement both and try out to be sure about this. So I would encourage you to do the same if you need that extra performance.
On the topic of using TreeNSearch for your application, well that depends on whether your application fits the type of problem that TreeNSearch was designed for. You can find this at the end of the README: "TreeNSeach is optimized to be run many times as points move very little between runs". Also, keep in mind that you cannot query TreeNSearch on a per-point basis and that it doesn't find n nearest neighbors, it only finds neighbors within a radius. So, in conclusion, if your problem is "I have a lot of points and I need the list of neighbors within a given search radius of every one of these points", then you can use TreeNSearch. Otherwise I am afraid it is not the tool you are looking for :)
Best regards.
José.

Thanks! I'll try it out.
For a photon map application, the photon map (millions of points) is built once and then queried thousands of times. So, TreeNSearch fits, I believe.

Sounds good!
As a final note, keep in mind that TreeNSearch only works when all the query points are known at the same time. In other words, you cannot query for neighbors for a single point (at least not efficiently).
The way I imagine this would work for you is to declare the points that make up your photon map as one point set and all the query points as another point set. Set TreeNSearch to search from the query points set into the photon map set. This way you get the neighbors for all the query points in one go.

Does it run on a GPU ?  Thanks.

TreeNSearch is exclusive to CPU.


API for Python: Non converged phonon band structure w/ ASE calculators

I'm finding that when I use the ASE to generate structures and calculate the forces using  ASE calculator interface for classical interatomic potentials, and then move it to Phonopy via the python API the phonon calculations are not converged, unless I use unreasonably large distance values. I am for sure using the ground-state (i.e., minimized unit cell) and the ASE units are eV, Angstrom, eV/Angstrom. Here is the code and outputs:
import ase
from phonopy import Phonopy
from phonopy.structure.atoms import PhonopyAtoms
...
calculator =  ase.calculators.emt.EMT() # units eV, eV/Angstrom

 # ase.Atoms object w/ min. to ground-state
structure =  ase_copper

phnpy_struct = PhonopyAtoms(
    symbols=structure.get_chemical_symbols(),
    positions=structure.get_positions(),
    cell=structure.get_cell(),
    masses=structure.get_masses(),
    )

phonons = Phonopy(
    phnpy_struct,
    supercell_matrix=[[6, 0, 0],
                      [0, 6, 0],
                      [0, 0, 6]],
    )
phonons.generate_displacements(distance=0.01)
sets_of_forces = []
supercells = phonons.get_supercells_with_displacements()

for i,d in enumerate(supercells):
    # Convert back to ASE
    d_ase = Atoms(symbols=d.get_chemical_symbols(),
        positions=d.get_positions(),
        cell=d.get_cell(),
        )
    d_ase.set_calculator(calculator)
    forces = d_ase.get_forces()
    sets_of_forces.append(forces)

sets_of_forces = np.array(sets_of_forces)
phonons.forces = sets_of_forces
phonons.produce_force_constants()

This runs with no issues and then running:
phonons.auto_band_structure(npoints=100,plot=True,write_yaml=True)

produces the following:

Importantly if I keep increasing  displacement, e.g.,distance=0.5, I get result:

and then final at distance=0.9 a converged band structure:

Question
This seems unreasonable values to use for harmonic approx method? Is there something fundamentally incorrect with what I'm doing?
Also, using a larger supercell doesn't appear to shift down the displacement distance required for convergence.
Example notebook
https://github.com/stefanbringuier/randomonium/blob/main/notebooks/ASE_Phonopy_Testing.ipynb

This issue is probably a question about usage of phonopy. I expect those questions to be sent to the phonopy mailing list, because people not only me can reply questions. Here, I hope that github issues is more likely used for the code development. I don't answer about ASE. I recommend to try produce_force_constants(fc_calculator='alm').

Just also to add, there are many working examples where phonopy and ase are used successfully together (e.g., see here for one of my example use cases with EMT calculator https://jageo.github.io/Advanced_Jobflow_Tutorial/tutorial.html#workflow-to-compute-phonons-vibrational-properties-of-crystals). Most likely, something is off with your structure conversion (cartesian vs. direct coordinates maybe?).

This issue is probably a question about usage of phonopy. I expect those questions to be sent to the phonopy mailing list, because people not only me can reply questions. Here, I hope that github issues is more likely used for the code development. I don't answer about ASE. I recommend to try produce_force_constants(fc_calculator='alm').

Agree, this is about use, so will email the mailing list (wasn't aware). Will also add kwarg. Thank you.

Just also to add, there are many working examples where phonopy and ase are used successfully together (e.g., see here for one of my example use cases with EMT calculator https://jageo.github.io/Advanced_Jobflow_Tutorial/tutorial.html#workflow-to-compute-phonons-vibrational-properties-of-crystals). Most likely, something is off with your structure conversion (cartesian vs. direct coordinates maybe?).

@JaGeo so your tip helped me find the source. It wasn't exactly the coordinates, but that the PBC weren't be set for the force calculation. Closing and will post on the mailing list from here on out for usage. Thank you.


Use GitHub mirrors instead of Gerrit

Fixes #35

@jeffw16  can you please help me in it?

I'm not sure what's going, but also please note that GitHub says there are merge conflicts that need to be resolved.

I have resolved the merge conflict this was due to merging of some pull requests after this pr was made

Can anyone suggest me what should I do for the tests to pass?
I think there may be space issue due to which the test fails

Thanks for the help Amal
These changes are not there in GitHub but are in Gerrit, so should I keep this line as it is or make this pr a draft untill these changes are not added to GitHub?


Improvements to the readme

Added instructions how to run the compiled demo with QEMU

In Step 5, the qemu command should reference the elisa-cluster-demo-platform-qemux86-64.ext4 file, not the agl-cluster-demo-platform-qemux86-64.ext4,
I'm just getting started here.  My pc hardware demands the latest software and that's causing a number of problems, so I installed a chrooted Ubuntu 20.04.  The instructions almost worked when I followed the instructions (as they were before this update).  For qemu, I used the instructions from the agl webiste (https://docs.automotivelinux.org/en/jellyfish/#0_Getting_Started/2_Building_AGL_Image/5_1_x86_Emulation_and_Hardware/#3-deploying-the-agl-demo-image), and used the command
runqemu tmp/deploy/images/qemux86-64/agl-demo-platform-qemux86-64.qemuboot.conf kvm serialstdio slirp publicvnc audio
The system ran and showed a login prompt on the command line.  No graphics were displayed, however.  After some further Xorg configuration, graphics programmes (like xterm) display properly, but the demo doesn't.
Sudip suggested copying the generated file(s) out of chroot and then using qemu from my Distro to run the image using the command from here.
Doing that, my qemu-system-x86_64 (version 5.2.0 (openSUSE Tumbleweed)) complains:

Could not access KVM kernel module: No such file or directory\n qemu-system-x86_64: failed to initialize kvm: No such file or directory (Note on the agl site says to just delete the -enable-kvm option.  Did that)
qemu-system-x86_64: -show-cursor: warning: The -show-cursor option is deprecated. Please add show-cursor=on to your -display options. \n qemu-system-x86_64: -show-cursor: warning: When using the default display you can use -display default,show-cursor=on
qemu-system-x86_64: warning: '-soundhw hda' is deprecated, please use '-device intel-hda -device hda-duplex' instead audio-legacy: Unknown audio driver `pa'
not knowing where "audio-legacy" 'pa' comes from, I gave up.
here is the last command I used:
john@eit:~/ELISA/agl/test> qemu-system-x86_64 -snapshot -device virtio-net-pci,netdev=net0,mac=52:54:00:12:35:02    -watchdog i6300esb    -netdev user,id=net0,hostfwd=tcp::2222-:22,hostfwd=tcp::2323-:23    -drive file=./elisa-cluster-demo-platform-qemux86-64.ext4,if=virtio,format=raw    -display default,show-cursor=on -usb -device usb-tablet -device virtio-rng-pci    -vga virtio -device intel-hda -device hda-duplex    -machine q35 -cpu kvm64 -cpu qemu64,+ssse3,+sse4.1,+sse4.2,+popcnt    -m 4096  -smp 4 -m 2048    -serial mon:stdio -serial null -kernel ./bzImage    -append 'root=/dev/vda rw  console=ttyS0 mem=4096M ip=dhcp oprofile.timer=1 console=ttyS0,115200n8 quiet '

Going back to the chrooted OS, I tried the command from these instructions.
Doing that, my qemu-system-x86_64 version 4.2.0 (Debian 1:4.2-3ubuntu6) complains:

Could not access KVM kernel module: No such file or directory\n qemu-system-x86_64: failed to initialize kvm: No such file or directory.  (Ditto above)
Noticed that the agl instructions said to install qemu-system-x86 and ovmf AND export OVFM_PATH=/usr/share/ovmf/OVMF.fd IT WORKED!

Going back to my distro test directory...

the current ovmf doesn't have a .fd file (the agl instructions say to use OVMF_CODE.fd under Arch and Fedora and OVMF.fd under Debian and Fedora)...

Conclusions:

Installing qemu and ovm as well as setting OVMF_PATH must be included in the instructions
qemu version 5 changes a number of paramter names - future work is coming
ovmf (whatever that is...) is going to change
I aliased python to python in .bashrc.  I don't know what the rest of the world should do, but Ubuntu does nothing....
somewhere there was an instruction to add ~/bin to PATH, but I had done that anyway
AGL_TOP and OVMF_PATH should be exported from the user's .bashrc
Presumably in a chroot system:

"xhost +" has to be called in the distro system before chroot is called
DISPLAY=:1 (I don't know why it's not 0. 0 just doesn't work) has to be exported from ~/.profile
the qemu command relies on more than one file, so one should copy the whole directory out to the distro when you want to run the demo there

You have to source the jellyfish/build/agl-init-build-env script to run runqemu.  Looking at the source code, there are a number of command-line options that would simplify calling the demo and are likely to be more robust when agl starts changing things.  I assume that it didn't work in my environment because graphics wasn't turned on.  I'll leave configuring the call to the agl experts....
This is the command that worked for me under Ubuntu 20.04:
qemu-system-x86_64 -snapshot -device virtio-net-pci,netdev=net0,mac=52:54:00:12:35:02
-watchdog i6300esb
-netdev user,id=net0,hostfwd=tcp::2222-:22,hostfwd=tcp::2323-:23
-drive file=./build/tmp/deploy/images/qemux86-64/elisa-cluster-demo-platform-qemux86-64.ext4,if=virtio,format=raw
-show-cursor -usb -device usb-tablet -device virtio-rng-pci -vga virtio -soundhw hda
-machine q35 -cpu kvm64 -cpu qemu64,+ssse3,+sse4.1,+sse4.2,+popcnt
-m 4096  -smp 4 -m 2048
-serial mon:stdio -serial null -kernel ./build/tmp/deploy/images/qemux86-64/bzImage
-append 'root=/dev/vda rw  console=ttyS0 mem=4096M ip=dhcp oprofile.timer=1 console=ttyS0,115200n8 quiet '


fix sensormodel bugs

[ ] is deploymentform2 used?
[ ] sensor in deployment is passed as query not foreign key?


Format code using 'cargo fmt'

Superseded by https://github.com/paritytech/libsecp256k1/pull/53
Really sorry wasn't merging this before. We only recently decided to start using rustfmt.


Battery type shows "Ternary Lithium" always

If you select "Li-ion", "FiFePo4" or "LTO" at the android app the BMS reports "Ternary Lithium" as battery type always.

Cannot be fixed.


If I say please will you tell me?

How should I find the right class / struct for the Key / Value of my TMap when getting it from a FMap_Mirror

please


Changed to v0.1.5

Pull Request Test Coverage Report for Build 8

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 7:
0.0%

Covered Lines:
24

Relevant Lines:
24

💛  - Coveralls


[Bug/Feature]: Arabic script letters in citation keys

Debug log ID
LFZ25M2F-refs-euc
What happened?
Hi! I originally posted this on Zotero forums, and was asked to open an issue for it here.
I hope someone can help me with this:
Something strange happens to the citation keys of my Zotero entries in Arabic script. Some of the Arabic letters are automatically converted to Latin characters, but some are converted to question marks.
My citation key formula is auth+year.
An example:
The article author is هناء زايد عباس
The citation key becomes this:
hn��z�ydaab�s2020
Why are some of the Arabic characters converted and others not? If I could access the convertion table for this and modify it, it would be great.
The characters not being read, are ا and ء .

LFZ25M2F-refs-euc

Would hnAzAydEbAs2020 be a reasonable romanization for هناء زايد عباس?

In this case, the alif, "ا" is rendered "A" and the ayn "ع" is rendered E, which works well (I didn't know the citation keys were case sensitive!), but the hamza, "ء" is left out.
The letter "ء" is often rendered either with unicode character 02BE ( ʾ ), or some kind of accent or backtick. Popularly, it is also romanized with the number 2. If it should be a letter, I suggest "x", because on the Arabic keyboard, the "ء" is in the same spot as the "x" on a qwerty keyboard.

I use jsingua for the romanization, do you think you can discuss with the author of that library what the best choices are? I don't speak/read any arabic.

Do you mean with the author of jslingua?

Yes. I am not in a position to judge whether the changes you want make broader sense. @kariminf will get you a much better dialogue.

Ok, I will ask him!

The letter "ء" is often rendered either with unicode character 02BE ( ʾ ), or some kind of accent or backtick. Popularly, it is also romanized with the number 2. If it should be a letter, I suggest "x", because on the Arabic keyboard, the "ء" is in the same spot as the "x" on a qwerty keyboard.

The issue is that ' is not a legal character in citation keys, so it gets discarded. If 2 or x is an equally valid romanization, wouldn't it be better to make that a feature of jslingua? I would prefer to keep discussions about romanization between native speakers/experts, I cannot judge what is and isn't suitable.

I'm now getting hnAzAyd`bAs2020 as the citekey, I could make that hnA2zAyd`bAs2020 if you think it's better, but I will have to take your word for it that this is a global preference qua romanization rather thanm your personal preference. I have no way to distinguish the two.

Hi,
Using "Buckwalter" transliteration هناء زايد عباس will be transliterated as hnA' zAyd EbAs
But names are a bit different to transliterate; in this case it is transliterated as:
hanaa zayed abbas (both alef and hamza are transliterated to "a")
The problem with Arabic names transliteration is there are 2 different systems: east (English-pronunciation) and west (French-pronunciation).
For example, ابو بكر can be transliterated into abu bakr (east) or abou bakr (west)
I will introduce names transliteration in jslingua in the coming version

And replacing ' with 2?

' is replaced by another a
instead of hana' we write hanaa

For most purposes transliteration is phonetically oriented, i.e. we try to represent how the words are to be read. But in this case, we need an orthographically oriented transliteration, where each character is represented by one or two Latin keyboard characters

There are many different transliteration systems for Arabic, there is not one or two. زايد can be rendered Zayed or Zayyed or Zayid etc. but in this case it doesn't matter, what matters is that it is possible to anticipate what the citation key for a reference will say. You cannot program ي to convert to "ye" because it will not apply to other words. "zAyd" is therefore a more appropriate orthographical romanization in this case.

I could make it hnA2zAydEbAs2020 if you think it's better, but I will have to take your word for it that this is a global preference qua romanization rather than your personal preference. I have no way to distinguish the two.

Most academic transliteration systems will use symbols for this letter that are not legal characters for citation keys. And using "a" for two different Arabic characters seems to be confusing and imprecise. So I'm just trying to suggest other options that make sense. I don't think there is a right or wrong answer here. I suggested "2" and "x" for the reasons I mentioned, and I think either of them would work fine.

This is exactly why I prefer that this kind of discussion is held between native speakers/experts.

Ok, so what happens now?

You tell me -- I see these comments about different romanizations for names, and I don't know whether or how to act on these. If the current romanization I showed above suffices for you, I'm willing to add that to the next release. But they may change depending on the follow-up from @kariminf.

I think, since the objective is just to avoid special characters such as ' by replacing them by another such as 2, in this case a simple mapping from Buckwalter's transliteration is enough. So, no need to change jslingua (even if I add names transliteration, Buckwalter will still as it is).
There are other characters which can be considered as special :
https://en.wikipedia.org/wiki/Buckwalter_transliteration

I don't see the ' character in the buckwalter row of the table? The only non-alphanum characters I see are * and $. I see ' in the Qalam row.

Under the table, there are hamza variants in addition to alef and harakat :

lone hamza: '

} is also disallowed in citation keys.

BBTCitKeysArabic.pdf

I mapped the convertion table in the current version. What is it based on?

https://www.npmjs.com/package/transliteration

� is the unicode replacement character and can be loosely understood as "unknown character".

It doesn't seem easy to find a package with only allowed transliteration characters.
I don't know whether this is useful, but it shows a use of number two for versions of the letter hamza: https://github.com/amasad/arabish
u'ء': '2',
u'أ': '2',
u'ؤ': '2',
u'إ': '2',
u'ئ': '2',
u'آ': '2',

From https://github.com/amasad/arabish:

And it's not that hard!
...
With a better training corpus and some simple tweaking to the rules we can get at least up to 80% accuracy of Yamli or similar services.

These two sentences seem to contradict one another. Surely if it's "not that hard", accuracy would be 100%.

Ok, then what about simply using "a" for "ا" and ignoring all versions of "ء" so that they are converted to nothing? The only problem for researchers working with Arabic texts are the replacement characters.

I don't know what about it -- as I said, I have no idea what is reasonable/desirable here.

All these suggestions are reasonable and desirable, but there will always be different opinions.

If there will always be different opinions, I'm stuck negotiating between these opinions about a matter of which I understand nothing. But if the two of you can reach a consensus, I'll take that for now, with the proviso that if a different opinion swings by, I may change the implementation.

ignoring all versions of "ء"

What does "all versions" mean here? Is it just "all occurrences" or are there indeed different versions of this character?

Under the table, there are hamza variants in addition to alef and harakat :
* lone hamza: '

I have no idea what hamza, alef and harakat mean.
If you guys can get me a simple mapping table to pre-apply before passing the content through jslingua and/or one that I can apply after the string is passed through jslingua, that is fine, but it can't depend on me understanding anything of what is going on in the romanization.

I still don't understand what is the problem here. Do you want a transliteration which is based on : Phonetics or Orthography ?
As I understood,  you want just an orthography-oriented transliteration. In this case, I suggest a simple solution :

Use jslingua (Buckwalter) to transliterate Arabic letters into Latin; then
Use the following python dictionary to map into a semi-phonetically oriented transliteration

{
    "'": "a",
    ">": "a",
    "<": "i",
    "&": "u",
    "}": "i",
    "|": "a",
    "{": "a",
    "`": "a",
}

I still don't understand what is the problem here. Do you want a transliteration which is based on : Phonetics or Orthography ?

I don't even know what this means. I'm trying to negotiate a solution that makes sense to people who speak the language (which is not me).

As I understood, you want just an orthography-oriented transliteration. In this case, I suggest a simple solution :
* Use jslingua (Buckwalter) to transliterate Arabic letters into Latin; then

* Use the following python dictionary to map into a semi-phonetically oriented transliteration

This I can do. @Serena-UB, A new build will drop here in 10 minute or so.

@Serena-UB the citekey would come out as hnAazAydEbAs2020.

Perfect, thank you both!

But what about * and $ in Buckwalter - will they work in citation keys?

But what about * and $ in Buckwalter - will they work in citation keys?

Yes, those will work.

I downloaded it, but strangely it works only for the one reference, not for other references. Am I doing something wrong?

I've been trying in various ways. The new transliteration now works only for references I download to Zotero through chrome, but not if I create a new reference and type in the author name myself.

Have you tried refreshing the keys?

I've been trying in various ways. The new transliteration now works only for references I download to Zotero through chrome, but not if I create a new reference and type in the author name myself.

That is strange. Please turn on debug logging in the Help menu, create such a new reference again, and then right-click that item and send a debug log.

Yes, I tried refreshing the keys.
If I download a reference, it works, even if I change the author name and retype it. But if I create a new reference, I get the old transliteration system
I already submitted a new debug log for it: STR2X8IX-refs-euc

The item doesn't have a language set. For implicit romanization, you need to set the language. You can also explicitly do auth.transliterate(ar) + year.

Ah, I see, it works now. Thanks!


Indentation

Continued from #60...
Note: this issue is not just for lists, everyone is free to suggest which elements and how they should be indented.

I think that CommonMark outlines how indentation should be processed. If we're violating any of those rules then issues could be opened to address specific ones. Closing since I don't think there is a specific issue here? (Please reopen if there is)


Add type annotations to fixtures

Helps with code completion in IDE

Basically concerns the test classes in base_tests, check whether there are some missing ones in the core.


[REFACTOR] process-monitoring: stop passing the bpmnVisualization instance to all functions

Introduce a class as for https://github.com/process-analytics/icpm-demo-2022/issues/29
Notice that the code only needs to access to the BpmnElementsRegistry instance, so we could pass it instead. See  #52

Done in #77