Display hostname on header

It includes the hostname in the header next to the logout button.
This will facilitate the recognition of pfsense is operating.

I had a similar feature in previous versions, so I brought it to the latest version.

Before this pull request can be accepted you must first sign a CLA as described at https://www.pfsense.org/about-pfsense/#cla. Please read for more details.

Thanks for you work. This feature has been added in another way (that does not consume menu real-estate.


Django 1.9 support

Django databrowse is affected most importantly by:

_meta changes:
https://docs.djangoproject.com/en/1.9/ref/models/meta/#migrating-old-meta-api
get_models and get_model are now in from django.apps import apps
URL patterns are deprecated, just use a list of url functions

@fmalina Pull request #21 has been merged into and claims to be fixing the issue for Django 1.9, could you try the latest from the master branch and let me know about it ?
If it fixes the issue, I'll be releasing a new version


BUG: token expire will return frontend server error

Describe the task
Sabina reports an error when she tries to grant Kevin access in FAM.

login to SPAR TEST
open new tab and login to FAM
get errors
refresh FAM
try again get errors
close the entire window and clear cache
come back to FAM
try to verify Kevin in FAM, but got the error
tried again and it worked
then tried Krista Copeland and got error

Login to SPAR and FAM both with IDIR
Check the PROD Cognito log, and found a jwt expired error. So when user logs in more than 5mins, and try to verify the user, we probably don't do an auto refresh of the token, so backend can't validate the token. But need to check when we auto refresh token.

Acceptance Criteria

[ ] Investigate when and where we auto refresh user tokens
[ ] Handle the situation when user token expired

Additional context
Investigation ticket

Solved as par of #1610


allow to forward 7-digit number if user set area code

allow to forward 7-digit number

Coverage remained the same at 40.169% when pulling b07b260dc886b80becb19b66a15914442af9f974 on embbnux:enhancement-webphone-forward into 13af5fe3101f988fe63cbc22e4a15a56dcd9ef8b on ringcentral:master.


Week 6 - 1.d: Seeding the database with example data

As a developer on the team, I want to make sure our deployed Chirp.Razor application on Azure App Services is always seeded with our example data, so that whenever we access our application, we expect to find at least the cheeps written by the authors in the DbInitializer.cs file that contains our data.
Acceptance Criteria
Using the file DbInitializer.cs to import example data into our Chirp.Razor application on application start.
Whenever we access our application, we expect to find at least the cheeps written by the authors in that file.
After creating our database context object, we can call the only method from DbInitializer.cs to seed the example data into our application.

dump.sql and schema.sql are currently not called by the program, and they expect a vastly different database architecture than what we actually have


Fix import versions direclty uuid

Hubspot uses request
request uses uuid
Which causes the following issue

@shahen94 which version of request are you using? Can you check with npm ls request?

@shahen94  Please also do npm ls uuid, as I suspect this is a bundler issue.    request  has been deprecated since Feb 2020 and is no longer maintained.  It depends on an old version of this module<EMAIL_ADDRESS>which does allow for direct import of v4.  However we deprecated direct imports subsequent to that version.
If request is trying to load v4 from a newer version of this module then I suspect there's something wrong with your node_modules installation, or with how your bundler/node is resolving modules.

@broofa @LinusU Just runned npm ls request
<EMAIL_ADDRESS>

So the only solution that I've found is to update uuid/package.json and mark vX exportable.

So you're editing the package.json at ./node_modules/uuid/package.json.  I.e the package.json for the [very old] uuid~3.4.0 module?
If so, then please note we're now on version 8.x, which is the version you're asking us to patch with this PR, and which is  a very different codebase than v3.  Even if we were to do a maintenance release for version 3.x, you'd have to get request to pick up that change in their dependencies to fix this issue.  But that's never going to happen because request is frozen and no longer maintained.
I'm actually still puzzled why you're seeing this problem.  It's not like there aren't 100K's of users out there that depend on request without issue.  What bundle/packager toolchain are you using?

Hi @broofa sorry for the delayed reply.
We're using webpack for this. if we'll not merge this PR, is there any way that I can monkey-patch the package locally ?

This problem may appear if you configure resolve.modules incorrectly.

This will let you patch any package in your node_modules on postinstall
https://github.com/ds300/patch-package


fix(runtime-core): unexpected props type inference failed

Close #1891

Maybe https://github.com/vuejs/vue-next/pull/1896 is better

It's weird and it still works correctly even though I do this:
DefaultFactory<T extends any ? D : T>

  default?: D | DefaultFactory<D extends any ? T : D> | null | undefined, // works
  default?: D | DefaultFactory<D extends any ? D : T> | null | undefined, // works
  default?: D | DefaultFactory<D extends any ? T : T> | null | undefined, // works
  default?: D | DefaultFactory<T> | null | undefined // doesn't work

closed in favor of #1896


Let client use from xstate import Machine

Thanks for this! It's my first published Python library, so thanks for helping smooth the rough parts 😅

Thanks for this! It's my first published Python library, so thanks for helping smooth the rough parts 😅

My pleasure :)

My pleasure :)

Anyway, you did great work :)

Anyway, you did great work :)


updated logviewsettings

Progress

Merged that latest changes from #865 and subsequent tweaks from master into this branch.
Prepared a first draft on how the new settings object for the LogView plugin could be used, see this comment for background info.

ToDo

Discuss and tweak the draft. If we're happy with it, proceed:
Add XML docs for LogViewSettings class and members.
Add XML docs to all added API in LogView.
(Squash-)Merge PR and applying the same principle when addressing the others linked in #855.

Ping @Barsonax, since I can't request a review from you.

Looks good. One thing is that I see you merged master into this branch. Its cleaner to rebase this branch on master as that prevents the merge commit from being generated. Rebasing simply puts the new commits of this branch on top of master so you end up with a cleaner history.

Can't approve since its my own PR so gonna to that by comment: Approved.
Merging this.


EfficientNet transfer learning - init with imagenet weights changes model

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
binary
TensorFlow version
2.14
Custom code
No
OS platform and distribution
Linux Ubuntu 22.04
Mobile device
No response
Python version
3.10
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
11.8
GPU model and memory
No response
Current behavior?
My current pipeline of:

Train a model using EfficientNet backbone with weights="imagenet"
Transfer a trained model weights to the prod environment
Initialize the production model with weights=None and load the previously learned weights
Compare the model's output between training and productions environment - they are different

Obviously, I expected the outputs to be the same. After LOTS of investigation, I found the culprit which is NOT mentioned anywhere in the documentation: github link.
Turns out, when the model is initialized with weights="imagenet" an additional rescaling layer is added. I understand the reason behind it (reproducing the Imagenet results) but the above situation proves how misleading the implementation is.
Why didn't I just save the whole model instead of transferring only the weights?
Well, because the model is trained with mixed precision on a GPU and transferring it this way to a production env which uses a CPU would make it unusable (nans all around).
What I would expect ideally:
The same model structure no matter which weights are loaded.
What I would expect to minimize the impact of this issue:
When the weights are loaded to production model, the mismatch between number of layers is detected and a warning is raised.
See test collab notebook with the code from below: link
Standalone code to reproduce the issue
import tensorflow as tf
import numpy as np
from keras.applications.efficientnet import EfficientNetB3, preprocess_input

img_shape = (300,300,3)
x = np.ones((1,) + img_shape)*255

trained_model = EfficientNetB3(weights="imagenet", include_top=False, input_shape=img_shape)
trained_model.save_weights("weights.h5")

prod_model = EfficientNetB3(weights=None, include_top=False, input_shape=img_shape)
prod_model.load_weights("weights.h5")

y_trained = trained_model.predict(preprocess_input(x), verbose=0)[0,:2,:2,:2]
y_prod = prod_model.predict(preprocess_input(x), verbose=0)[0,:2,:2,:2]

print(f"Trained model output (initialized with imagenet weights):\n{y_trained}")
print(f"Production model output (initialized with 'None' weights):\n{y_prod}")

Relevant log output
Trained model output (initialized with imagenet weights):
[[[-0.27523986 -0.25020567]
  [-0.26998693 -0.27772853]]

 [[-0.2728562  -0.27376127]
  [-0.24821427 -0.23993279]]]
Production model output (initialized with 'None' weights):
[[[-0.2746129   0.89038897]
  [-0.25032952  0.1611695 ]]

 [[-0.1986168   0.0976367 ]
  [-0.17414406 -0.26978934]]]

@google-admin can you assign someone manually to this case? It seems the bot missed it after changing the title?

@maksym33 ,
Apologies for the bug in auto-assignment.I have replicated the issue and confirm difference in outputs before and after reloading the model as per attached gist. Need to check the root cause. Thanks.

Hi @maksym33 ,
Apologies for the dealy, I tried to run your code on Colab using TF v2.17.0, nightly and faced the same issue. Please find the gist here for reference.
Please post this issue on keras-team/keras repo. as this issue is more related to keras
Thank you!


Handle federated followers-only posts, currently treated same as mentioned-only

Posted a toot on Mastodon 4.0.x (Hometown, technically) with "Followers" visibility. No users were mentioned in it:

It appears in the Takahē database with Visibility = Mentioned (i.e., a "DM").

(And it does not appear in the timeline of a user who follows the author, because they weren't mentioned.)
Expected: the post appears in the database with Visibility = Followers.
Takahē 0.9.0 Docker image.

Visibility remains Mentioned after the state changes to fanned_out.
For more fun, if the original poster on Mastodon boosts their followers-only post, it does appear in the Takahē user's timeline, with the client's DM bubble around it:

I tested all the post types that federate: public (works), unlisted (works), mentioned-only (works). Only followers-only posts get miscategorized as "Mentioned". And, reading the code, I cannot figure out why!

Oh right, I almost forgot I had put this shortcut in a while back - I wasn't seeing a lot of followers-only posts back then so I just decided to make it default to the even more restricted one.
Let me see if I can get a few test messages from my own server and get this working. We still don't fetch follower objects but we can now serve them, at least.

Alright, fix is into main to make it work both ways - the only thing I noticed is that Mastodon may not respect it as a followers-only post until it refreshes the identity to see that it has followers/following URIs. If you need to force that, just make a tiny edit to your profile and Takahē will ping all follower servers to make them pull it again.


search suggestion and context menu

Is there a reason that search suggestion from bookmarked and opened tabs are disabled?
I am using the following:
browser.urlbar.maxRichResults = 8
browser.urlbar.suggest.bookmark = true
browser.urlbar.suggest.openpage = true
Just can't see any privacy problems here, only productivity improvements.
Also accessibility.typeaheadfind = true seems nice productivity improvement.
For dom.event.contextmenu.enabled I am not sure about privacy/security (?), but some pages (outlook web access for example) provides additional control over context menu.

I also use the 1st 3 prefs, not a fan of the 'typeaheadfind' (prefer CTRL+F), and agree that 'dom.event.contextmenu.enabled' needs some looking into regarding privacy/security.
We could create a section for 'physical location / spying / shoulder surfing, etc VS productivity/convenience/usability", and comment them out by default so users can enable them if they need protection against cam-surveillance, shoulder surfing, etc.
I imagine a majority of users are in a "safe environment" and would prefer the productivity improvements, etc.
IMO we can add accessibility.typeaheadfind as "enable typeaheadfind = true" and comment it out, under the personal section.

0800 says in the header "Not ALL of these are strictly needed, some are for the truly paranoid". I don't want to go overboard, but we have a [SETUP] tag. If it doesn't cause site breakage (so no downsides), then it gets used to enhance security or privacy (as a general rule) - remember the old purpose: to increase Sec + Privacy and reduce FP while remaining functional and with as little site breakage as possible.
We already had a 12bytes user ask about getting back his urlbar suggestions, hence the commit to 0808. And I am thinking of a wiki subsection of "quick fixes" (not going to use the word fix) such as  Restore video settings etc, where the intro paragraph says the pros and cons, and underneath it lists the items to look at. Video would be a good one.
Yes, 0800 could do with a little rejig, I mean jumplists, taskbar previews, etc - yeah maybe these could be grouped a little better together
// 0850: if in an insecure environment such as an open office with shoulder surfers.. blah blah [SETUP]
// 0850a: ... or 0851 .. I can see about 8 or 9 that fit that

I still like the idea of making this overall more modular by using a preferences folder.
But we would only use a separate file IF we can really isolate certain prefs for a certain feature, SB + TP come to mind.
We could release with safebrowsing-trackerprotection.js.prefs and without needing to edit anything, users could just rename the file to disable SB + TP.
Since you plan to enable SB+TP by default anyway, I think this would be a neat way to handle it.

Restore video settings

If we had multiple files, we could even consider using 'pref' instead of 'user_pref' for certain files (fe media)
I know it has a lot of downsides, but one major advantage is that the file could simply be renamed or deleted and the original defaults would be restored.

@earthlng .. man, you're up early today. Will dump this on you - looking for feedback. This is not a final, but a very rough first draft. It's a complete revamp of 0800. Note 0604 is moved into this 0800s. Ignore numbers, we can tidy that later. 0801-0809 so far = all highly recommended to leave alone for privacy ... those 0850+ = relax for more convenience eg if sole user of machine, not in an open office etc. I also tried to numerically order stuff together - eg 1850a+b+c. I plan to add formfill days pref and download history retention pref (and will test them as well).  Is 0809 a LIVE pref or just a suggestion to use the search engine from the urlbar? Thoughts? Class ...Discuss!

/* note remove 0604 which moves to 0800 ***/

/*** 0800: LOCATION BAR / SEARCH BAR / SUGGESTIONS / HISTORY / FORMS / DOWNLOADS [SETUP]
     If you are in a private environment (no unwanted eyeballs) and your device is private (you
     are the only user), and the device is secure (locked, encrypted, forsensic proof), then items
     0850 and above can be relaxed in return for more convenience and functionlity. Likewise, you
     may want to check the items cleared on shutdown in section 2800.
     [NOTE] The urlbar is also commonly refered to as the location bar and address bar
 ***/
user_pref("ghacks_user.js.parrot", "0800 syntax error: the parrot's ceased to be!");
/* 0801: disable location bar using search - PRIVACY
 * don't leak typos to a search engine, give an error message instead ***/
user_pref("keyword.enabled", false);
/* 0802: disable location bar domain guessing - PRIVACY/SECURITY
 * domain guessing intercepts DNS "hostname not found errors" and resends a
 * request (eg by adding www or .com). This is inconsistent use (eg FQDNs), does not work
 * via Proxy Servers (different error), is a flawed use of DNS (TLDs: why treat .com
 * as the 411 for DNS errors?), privacy issues (why connect to sites you didn't
 * intend to), can leak sensitive data (eg query strings: eg Princeton attack),
 * and is a security risk (eg common typos & malicious sites set up to exploit this) ***/
user_pref("browser.fixup.alternate.enabled", false);
/* 0803: display all parts of the url in the location bar - helps SECURITY ***/
user_pref("browser.urlbar.trimURLs", false);
/* 0804: limit history leaks via enumeration (PER TAB: back/forward) - PRIVACY
 * This is a PER TAB session history. You still have a full history stored under all history
 * default=50, minimum=1=currentpage, 2 is the recommended minimum as some pages
 * use it as a means of referral (eg hotlinking), 4 or 6 may be more practical ***/
user_pref("browser.sessionhistory.max_entries", 4);
/* 0805: disable CSS querying page history - CSS history leak - PRIVACY
 * [NOTE] this has NEVER been fully "resolved": in Mozilla/docs it is stated it's
 * only in 'certain circumstances', also see latest comments in the bug link
 * https://dbaron.org/mozilla/visited-privacy
 * https://bugzilla.mozilla.org/show_bug.cgi?id=147777
 * https://developer.mozilla.org/en-US/docs/Web/CSS/Privacy_and_the_:visited_selector ***/
user_pref("layout.css.visited_links_enabled", false);
/* 0806: disable displaying javascript in history URLs - SECURITY ***/
user_pref("browser.urlbar.filter.javascript", true);
/* 0807: disable LIVE search suggestions - PRIVACY
 * This setting is under Options>Search>Provide search suggestions ***/
user_pref("browser.search.suggest.enabled", false);
/* 0808: disable LIVE search suggestions in the urlbar (requires 0807 = true) - PRIVACY
 * This setting is under Options>Search>Show search suggestions in location bar results ***/
user_pref("browser.urlbar.suggest.searches", false);
/* 0809: disable LIVE search location bar opt-in prompt (FF41+) - PRIVACY ***/
user_pref("browser.urlbar.userMadeSearchSuggestionsChoice", true);

/* 0850a: disable location bar autocomplete ***/
user_pref("browser.urlbar.autocomplete.enabled", false);
/* 0850b: disable locationbar dropdown ***/
user_pref("browser.urlbar.maxRichResults", 0);
/* 0850c: disable types of location bar suggestions
 * These settings are under Options>Privacy>Location Bar. If you wish to enable any of these suggestions,
 * then also make sure 0850a (enable autocomplete) and 0850b (location bar dropdown) are at default ***/
user_pref("browser.urlbar.suggest.history", false);
user_pref("browser.urlbar.suggest.bookmark", false);
user_pref("browser.urlbar.suggest.openpage", false);
/* 0850d: disable location bar autofill
 * http://kb.mozillazine.org/Inline_autocomplete ***/
user_pref("browser.urlbar.autoFill", false);
user_pref("browser.urlbar.autoFill.typed", false);
/* 0850e: disable one-off searches from the addressbar (FF51+)
 * http://www.ghacks.net/2016/08/09/firefox-one-off-searches-address-bar/ ***/
user_pref("browser.urlbar.oneOffSearches", false);

/* 0860: disable search and form history
 * Under Options>Privacy> if you set Firefox to "use custom settings" there will be a
 * setting called "Remember search and form history".
 * You can clear formdata on exiting Firefox (see 2803) ***/
   // user_pref("browser.formfill.enable", false);
/* 0861: disable saving form data on secure website
 * For convenience & functionality, this is best left at default true.
 * You can clear formdata on exiting Firefox (see 2803) ***/
   // user_pref("browser.formfill.saveHttpsForms", false);
/* 0862: disable browsing and download history
 * Under Options>Privacy> if you set Firefox to "use custom settings" there will be a
 * setting called "Remember my browsing and download history"
 * You can clear history and downloads on exiting Firefox (see 2803) ***/
   // user_pref("places.history.enabled", false);

/* 0870: disable Jumplist (Windows7+) ***/
user_pref("browser.taskbar.lists.enabled", false);
user_pref("browser.taskbar.lists.frequent.enabled", false);
user_pref("browser.taskbar.lists.recent.enabled", false);
user_pref("browser.taskbar.lists.tasks.enabled", false);
/* 0871: disable taskbar preview ***/
user_pref("browser.taskbar.previews.enable", false);

^^ am thinking even
/*** 0800: LOCATION BAR / SEARCH BAR / SUGGESTIONS / HISTORY / FORMS / DOWNLOADS [RECOMMENDED] ***/
blah blah

/*** 0850: LOCATION BAR / SEARCH BAR / SUGGESTIONS / HISTORY / FORMS / DOWNLOADS [SETUP]
     If you are in a private environment (no unwanted eyeballs) and your device is private
     (you are the only user), and the device is secure (locked, encrypted, forensic proof),
     then these can be relaxed in return for more convenience and functionality. Likewise,
     you may want to check the items cleared on shutdown in section 2800.
     [NOTE] The urlbar is also commonly referred to as the location bar and address bar
 ***/

Love it, love it, love it!
0809 - with browser.search.suggest.enabled=false the prompt never shows up. if it's true and browser.urlbar.suggest.searches=false, it'll show the prompt

Would you like to improve your search experience with suggestions? Learn more... [yes/no]

We can either ditch it completely, add it to 0807 or leave it as a separate item. I don't care.
now for some nitpicking and other ideas...
I would still add the test-link for 0805 so that users can see for themselves that it is still a problem.
With a note behind the test-link that the test only works in a normal window.
Instead of This setting is under Options>Search>Provide search suggestions we could use [Setting] Options>Search>Provide search suggestions, because it's not really under, it's that setting (Provide search suggestions)
0804 - I have mine set to 10 for convenience and use the history.length-spoofing userscript.
Idk if we want to move 0804 to 0850+ too and recommend and link to our userscript instead. or not. just and idea.
yeah, not much more to say, I love it - great job!
ps. download history retention pref - I don't think does anything but feel free to test yourself.

history.length-spoofing userscript is security by obscurity. The pref actually is the security. I've actually learnt to live with 4 .. been like 3 times in the last year I went damn .. but then I still have the full history under the menu>History (but that's all tabs) ... might bump it to 6 or 10, we'll see.

yeah, it was more food-for-thought, not something I feel strongly about. Since we want to make this more appealing to a broader community, a slightly larger "default" value maybe wouldn't hurt, and the truly paranoid can still set it to 2 or whatever.

@Roman-Nopantski just in case you didn't know with the markdown syntax you can tell GitHub the language in the code block:
var foo = "bar"

@Roman-Nopantski just in case you didn't know with the markdown syntax you can tell GitHub the language in the code block:
```
/* comment */
user_pref("foo", "bar");
```
/* comment */
user_pref("foo", "bar");

```js
/* comment */
user_pref("foo", "bar");
```
/* comment */
user_pref("foo", "bar");

IMO I think it's easier on the eyes.

Just saying .. my rubber hose and $5 wrench (used on the perp or the perp's loved one(s)) and/or some zero-day exploits and/or social engineering and/or future brute force capabilities beats this. I would of course take the easy route if possible, and just secure the device when it is unlocked (sniper? distraction, ninja ... ) .. Class, discuss!
If you are in a private environment (no unwanted eyeballs) and your device is private
(restricted access), and the device is secure when unattended (locked, encrypted, forensic
hardened), then...

@earthlng added this

browser.formfill.expire_days is interesting. I am not 100% sure how/when this is applied. The form data database has an expiry date field (not sure on creation date?) , so obviously that field is set as each entry is added when you fill in form data. But I am not sure if there is a periodic or set routine that updates expiry dates if this pref value is changed. See https://dxr.mozilla.org/mozilla-central/source/toolkit/components/satchel/FormHistory.jsm. Since the default is 180 days, then I guess it is recommended that users clear it out first? IDK .. am I overthinking this?
Also, interestingly, this is a very good way to control just form data, as other prefs are bundled. Eg browser.formfill.enable and privacy.clearOnShutdown.formdata are both Search+Form.
So if you allow browser.formfill.enable but set browser.formfill.expire_days to zero , then this is the combination to only allow search history. BUT - someone needs to test it. I am not 100% sure that a value of zero means no form data history. I am also sure this does not affect saved logins/passwords, but, I haven't checked.
Man .. this is so close to being finished. I even have a ref/source for my ninja comment:
#Required reading: https://xkcd.com/538/

I can test this. Will try today, but to be sure just give me a day or two, if today something come in between. ;)

I'm going to try and wrap this tonight. I want to get things finished. By all means test, and we can always edit the commit, but I will probably test myself anyway.

OK... first observations.
Testing is done with a clean default profile.
Installed SQLite Manager extension.
Connected to moz_formhistory table in formhistory.sqlite database.
Defaults:
browser.formfill.enable = true
browser.formfill.expire_days = 180
Testing with defaults... no surprises here... obviously. :)
Testing with:
browser.formfill.enable = true
browser.formfill.expire_days = 0
This one remebers formfills even when browser is restarted. My first thought would be that formfills data are forgotten instantly or at least after restart on a first day, but no.
Actually browser.formfill.expire_days = 0 does exactly nothing.
Testing with:
browser.formfill.enable = false
browser.formfill.expire_days = <doesnt matter>
Formfill data are not written into database as expected and data are forgotten instantly.
But all data from before remains even with restart.
Expected that privacy.clearOnShutdown.formdata = true would clean it up, but no.
Now I need to test it with:
browser.formfill.expire_days = 1 and will see tommorow.
Do you have any additional test cases in mind?
If so, just throw it here and I will test it.
Cheers

added this

:thumbsup:
browser.formfill.expire_days - it looks like the code is never actually triggered except in a test-file.
First of all, idk what this line is good for, there's exactly one line of code that uses formhistory-beforeexpireoldentries
sendNotification("formhistory-beforeexpireoldentries", expireTime);

The function expireOldEntries is only called once as far as I can see:
FormHistory.expireOldEntries();
and is only triggered when the observer observes formhistory-expire-now, but the problem is only 1 test ever sends that trigger: formhistory-expire-now
Maybe I'm missing something. Help? ;)

Appendix C: Test Sites - we should also list the URLs visibly -> more info, more better ;)

Appendix C: Test Sites - we should also list the URLs visibly -> more info, more better ;)

Are u serious? People can mouse over

@earthlng Should I do any additional tests for browser.formfill.expire_days?

Here's what I wanted to test:

1 browser.formfill.expire_days (default 180 days). We leave FF history settings at default, that is we collect and save search and form history and do not clear it per session..

Get some form data, it should have an expiry date set 180 days from date(now)
Change the value to 90 days - does the existing data get updated - check if it happens on a restart
Clear all data, change the value to 1, check data
Clear all data, change the value to 0, is any data collected

2 browser.download.manager.retention - same deal. We leave places.history.enabled (which is a joint browsing + download` history) as default, and do not clear session history. I want to see if when this is set to zero, if DL history is retained at all.

These two above could:

allow the combined Search+Form on but fine tune Form
allow the combined Browsing+Download but then turn off the DL

Does browser.urlbar.oneOffSearches (which is in the setup section >0850) require browser.urlbar.suggest.searches (which is in the recommended section <0850)

That was it. Clean those up and and I can post a final for nits

Sweet ..FYI: FT DeepDark author is going to fix the about:networking page. I emailed him earlier, then sent him both copies of the chrome://mozapps/skin/aboutNetworking.css files from each applied theme. He emailed me back. Nice guy.

3 - no. FF52 also seems to have fixed the dependency on keyword.enabled. And it does even work with browser.urlbar.maxRichResults=0 contrary to what @pyllyukko has stated here

0819: disable one-off searches from the addressbar (FF51+)
Covered by browser.urlbar.maxRichResults

I really like it though and IMO we should ship with browser.urlbar.oneOffSearches;false commented out.

Are u serious? People can mouse over

Yes I'm serious. Having to mouse over is just not as pleasant as seeing it all right away.
If it's too much work I'm happy to do the changes myself, if you approve.

Current is clean. Look, I'm a fan of functionality over form.   Section 3 sucks as it adds double the lines and basically halves the items seen in a glance. Are you suggesting using smelly tables, ewww :).

tables would work and are easy to do in markdown. using multiple   should also work but sucks.
2. with the link on the "name" and the url in brackets would be good enough for me

Panopticlick https://panopticlick.eff.org/

I really don't care, sorry. I just want to get these tasks finished, do a 52 release, finish the wiki pages I need and finalize a simple basic readme, so I can get Martin to an article. 1600 DLs in 5 days .. when we did v11 ... meanwhile, so far 30 odd unique visitors to the github ghacks user.js in 3 weeks - I'm getting antsy.  So in future, we don't have 50 commits a friggin week. So I really do not care about the wiki links looks. They can stay as they are. Clean and effective.
Meanwhile, everything we have done and is unique etc is being "caught up on" and "copied/imitated/inspiring" (and look, we don't have a copyright or monopoly on any of this, that's not what I mean).
nodiscc, bless his heart, has gone ballistic - adding section headers, regrouping items, adding a lot of missing prefs (well pyllyukko is, which is awesome, because sharing info and helping both of us get the best is great) and reviewing some via the monster diff (in other words, getting up to speed), reviewing large sections (sometimes it feels like I'm being used - 10 days after we start our 1600 section review, so does nodiscc). And they're revamping their readme as well. And after takling to pyllyukko on irc, and mentioning my plans to do several versions/branches, now pyllyukko is steaming ahead with that (obviously my chat reminded him to do it, it's not anyone's original idea, and he had obviously thought about it before). Suddenly tags are being added, and a standardized formatting implemented as well (for some automation of the readme), ..  Sure, some info was gleaned from pykky's but I always gave his repo top front and center acknowledgement, despite the fact that I hardly used it.  This is OUR work, and I feel (probably irrationally), that after a year of talking about coming here, we do, only to be upstaged by the time we "launch".
Yes, I know, I'm not thinking rationally. I respect pyllyukko, always will.  I feel my baby (and yours too earthlng for the last 12 months diff and stuff) is being siphoned off. 2 years of almost every day testing stuff. Some days spending hours on it. All to simply be looked at as an imitation of the FIRST repo here - because by definition, we only just "launched". People will always think PK came first and we raped him (terrible words, my bad).
So I don't care about the wiki hyperlinks.

@Roman-Nopantski

did all steps... it looks like browser.formfill.expire_days does nothing, the form data is collected when value is 0,1,90,180... restart does nothing to timestamps, existing data is not updated in any case
didn't understand what you wanted exactly here :(

Wow, I don't even know what to say. I don't understand where this is coming from.  And who is looking at this as some kind of imitation? I'm not the author of this project so I don't go around promoting it in all kinds of places and maybe that's why I don't see any of those negative comments.
Honestly I don't even care if only 5 or 10 people would be using this. I would be looking at the new and changed prefs anyway, so I might as well share them here and we can profit off each others work.
If some like-minded people find this useful, great - and if not, that's totally fine with me too.
I know that you appreciate the little input and help I provide and that's enough for me to stick around.
 This is now on github and if they want they can look at the comments + commits here and decide if they want to also include/change/deprecate some prefs in their user.js. And we can do the same.

And it even works with browser.urlbar.maxRichResults=0 contrary to what @pyllyukko has stated here

maybe they will see that and react to it, maybe not - why should I care?! Let them do their thing and we do our thing.

Why do we disable 2 of the 10 modern ciphers because of a theoretical future possibility but at the same time allow 2 older ciphers for compatibility reasons, with one of them also using AES128, and both being arguably weaker than the 2 modern ones?

I asked that question in another thread and none of them ever replied, so since then I realized that we are on our own here. Sure, it would be great if we could all get along and help each other out, but we can't force anyone to do anything, and tbh their input here has been minimal to non-existent, so yeah - fuck it.

@crssi

thanks for testing and confirming that. Looks like the code I looked at wasn't lying :)
I don't either. I can't speak for Pants, but I for one appreciate your inputs, feedback and testing a lot! thx

@earthlng
Hey, its a least I can do. And yes, your sight is just fine. :)
I would like to spare you as much time I can, since I am aware how much you are already investing in it.
It is me who needs to thank you guys.

I am a bit puzzled why privacy.clearOnShutdown.formdata = true does not clear formdata from moz_formhistory table in formhistory.sqlite database. Is this a FF bug?
Or its something wrong with my test observations? It could be latter, will retest it sometime today.

@crssi If that's the case we (earthlng, myself etc) need to replicate the same issue. [If it happens in 52, then we should also test on 53/54 to see if it still happens]. FWIW, my working pimped 52, the table moz_formhistory is empty - I allow form data to be remembered, I clear on shutdown. I just opened a new session 30 minutes ago.
PS: Thanks for your testing on the browser.formfill.expire_days (and earthlngs dxr code inspection) - maybe its partially there for ESR, but now that too has moved to a new cycle. Anyway, don't care - good, I can drop this pref.
PPS: Thanks @earthlng re question about browser.urlbar.oneOffSearches - now I know I don;t need to do anything else to it to explain if it breaks, cuz it won't
Just leave me with browser.download.manager.retention to finish off

browser.download.manager.retention does nothing... it looks like deprecated.

I also stand behind what I said, but in a way its complimentary that our arrival has spurred activity and imitation (either deliberate or inspired or whatever, zero fucks). None of our ideas are original, no one owns anything - it just grates that everything I have as a point of difference is being slowly eroded. That said ..  End of story. I give ZERO fucks.

@crssi Thanks. I'll replicate your delete retention test later and look in the DXR as well.

OK, so I had a look at the DXR. The browser.download.manager.retention is, AFAIK, is/was only for the downloads panel (that's the slidey down panel you get when you click the arrow icon). History is still retained. Just read this 5 year old ticket - and its all to do with some migration back in FF10 to 13 with a new DL's panel etc, blah blah.
Here's the relevant section: https://dxr.mozilla.org/mozilla-central/source/toolkit/components/downloads/nsDownloadManager.cpp#1044 .. and the pref is used on line 1052 (PREF_BDM_RETENTION was earlier set to the browser.download.manager.retention value). This is the only code that uses it.
I'm going to have a play because well, why is this code still around.

final patch for nits

/* note remove 0604 which moves to 0800 ***/

/*** 0800: LOCATION BAR / SEARCH BAR / SUGGESTIONS / HISTORY / FORMS / DOWNLOADS [SETUP]
     If you are in a private environment (no unwanted eyeballs) and your device is private
     (restricted access), and the device is secure when unattended (locked, encrypted, forensic
     hardened), then items 0850 and above can be relaxed in return for more convenience and
     functionality. Likewise, you may want to check the items cleared on shutdown in section 2800.
     [NOTE] The urlbar is also commonly referred to as the location bar and address bar
     #Required reading: https://xkcd.com/538/
 ***/
user_pref("ghacks_user.js.parrot", "0800 syntax error: the parrot's ceased to be!");
/* 0801: disable location bar using search - PRIVACY
 * don't leak typos to a search engine, give an error message instead ***/
user_pref("keyword.enabled", false);
/* 0802: disable location bar domain guessing - PRIVACY/SECURITY
 * domain guessing intercepts DNS "hostname not found errors" and resends a
 * request (eg by adding www or .com). This is inconsistent use (eg FQDNs), does not work
 * via Proxy Servers (different error), is a flawed use of DNS (TLDs: why treat .com
 * as the 411 for DNS errors?), privacy issues (why connect to sites you didn't
 * intend to), can leak sensitive data (eg query strings: eg Princeton attack),
 * and is a security risk (eg common typos & malicious sites set up to exploit this) ***/
user_pref("browser.fixup.alternate.enabled", false);
/* 0803: display all parts of the url in the location bar - helps SECURITY ***/
user_pref("browser.urlbar.trimURLs", false);
/* 0804: limit history leaks via enumeration (PER TAB: back/forward) - PRIVACY
 * This is a PER TAB session history. You still have a full history stored under all history
 * default=50, minimum=1=currentpage, 2 is the recommended minimum as some pages
 * use it as a means of referral (eg hotlinking), 4 or 6 or 10 may be more practical ***/
user_pref("browser.sessionhistory.max_entries", 10);
/* 0805: disable CSS querying page history - CSS history leak - PRIVACY
 * [NOTE] this has NEVER been fully "resolved": in Mozilla/docs it is stated it's
 * only in 'certain circumstances', also see latest comments in the bug link
 * [TEST] http://lcamtuf.coredump.cx/yahh/ (see the github wiki APPENDIX C: Test Sites on how to use)
 * https://dbaron.org/mozilla/visited-privacy
 * https://bugzilla.mozilla.org/show_bug.cgi?id=147777
 * https://developer.mozilla.org/en-US/docs/Web/CSS/Privacy_and_the_:visited_selector ***/
user_pref("layout.css.visited_links_enabled", false);
/* 0806: disable displaying javascript in history URLs - SECURITY ***/
user_pref("browser.urlbar.filter.javascript", true);
/* 0807: disable search bar LIVE search suggestions - PRIVACY
 * [SETTING] Options>Search>Provide search suggestions ***/
user_pref("browser.search.suggest.enabled", false);
/* 0808: disable location bar LIVE search suggestions (requires 0807 = true) - PRIVACY
 * Also disable the location bar prompt to enable/disable or learn more about it.
 * [SETTING] Options>Search>Show search suggestions in location bar results ***/
user_pref("browser.urlbar.suggest.searches", false);
user_pref("browser.urlbar.userMadeSearchSuggestionsChoice", true); // (FF41+)
/* 0850a: disable location bar autocomplete ***/
user_pref("browser.urlbar.autocomplete.enabled", false);
/* 0850b: disable location bar dropdown
 * This controls the maximum number of entries that can appear in the location bar dropdown.
 * Zero completely disables it. If you want ANY dropdown functionality, this must be changed ***/
user_pref("browser.urlbar.maxRichResults", 0);
/* 0850c: disable location bar suggestion types
 * [SETTING] Options>Privacy>Location Bar.
 * [NOTE] If you wish to enable these suggestions, make sure 0850a and 0850b are at default ***/
user_pref("browser.urlbar.suggest.history", false);
user_pref("browser.urlbar.suggest.bookmark", false);
user_pref("browser.urlbar.suggest.openpage", false);
/* 0850d: disable location bar autofill
 * http://kb.mozillazine.org/Inline_autocomplete ***/
user_pref("browser.urlbar.autoFill", false);
user_pref("browser.urlbar.autoFill.typed", false);
/* 0850e: disable location bar one-off searches (FF51+)
 * http://www.ghacks.net/2016/08/09/firefox-one-off-searches-address-bar/ ***/
user_pref("browser.urlbar.oneOffSearches", false);
/* 0860: disable search and form history
 * [SETTING] Options>Privacy>History>Custom Settings>Remember search and form history
 * [NOTE] You can clear formdata on exiting Firefox (see 2803) ***/
   // user_pref("browser.formfill.enable", false);
/* 0861: disable saving form history on secure websites
 * For convenience & functionality, this is best left at default true,
 * especially as the web moves more and more to encrypted services
 * You can clear form history on exiting Firefox (see 2803) ***/
   // user_pref("browser.formfill.saveHttpsForms", false);
/* 0862: disable browsing and download history
 * [SETTING] Options>Privacy>History>Custom Settings>Remember my browsing and download history
 * [NOTE] You can clear history and downloads on exiting Firefox (see 2803) ***/
   // user_pref("places.history.enabled", false);
/* 0870: disable Windows jumplist ***/
user_pref("browser.taskbar.lists.enabled", false);
user_pref("browser.taskbar.lists.frequent.enabled", false);
user_pref("browser.taskbar.lists.recent.enabled", false);
user_pref("browser.taskbar.lists.tasks.enabled", false);
/* 0871: disable Windows taskbar preview ***/
user_pref("browser.taskbar.previews.enable", false);

bump .. is this good to go?

bump .. is this good to go?

let's see... I'll re-post parts of my previous comment here re: nits

Pants: Is 0809 a LIVE pref or just a suggestion to use the search engine from the urlbar?

Earthlng: with browser.search.suggest.enabled=false the prompt never shows up. if it's true and browser.urlbar.suggest.searches=false, it'll show the prompt
Would you like to improve your search experience with suggestions? Learn more... [yes/no]
We can either ditch it completely, add it to 0807 or leave it as a separate item. I don't care.

You've since added browser.urlbar.userMadeSearchSuggestionsChoice to 0808.
I propose something like this (as a format I'd prefer in some cases, going forward)
/* 0807: disable search bar LIVE search suggestions - PRIVACY ***/
user_pref("browser.search.suggest.enabled", false); // Options>Search>Provide search suggestions
user_pref("browser.urlbar.suggest.searches", false); // Options>Search>Show search suggestions in location bar results
user_pref("browser.urlbar.userMadeSearchSuggestionsChoice", true); // (FF41+)

Because the titles are essentially the same for both 0807+0808 anyway

I would still add the test-link for 0805 so that users can see for themselves that it is still a problem.
With a note behind the test-link that the test only works in a normal window.

You've added the testpage with a note. If I'd change anything it'd be to shorten that line, fe.
[TEST] http://lcamtuf.coredump.cx/yahh/ (see github wiki APPENDIX C on how to use)
Optionally also drop the "on how to use"

Instead of This setting is under Options>Search>Provide search suggestions we could use [Setting] Options>Search>Provide search suggestions, because it's not really under, it's that setting

All done. Glad you liked it.

0804 - I have mine set to 10 for convenience and use the history.length-spoofing userscript.

you changed it to 10, nice. Do we want to include a link to the userscript?

Or you could add another line to the first version:
/*** 0850: browser.urlbar.* convenience/functionality vs privacy/paranoia ***/

I still think this would be nice - it would slightly separate the 0850+ lines a bit better visually.
user_pref("browser.urlbar.userMadeSearchSuggestionsChoice", true); // (FF41+)
/* 0850: browser.urlbar.* convenience/functionality vs privacy/paranoia ***/
/* 0850a: disable location bar autocomplete ***/
   // user_pref("browser.urlbar.autocomplete.enabled", false);

Most importantly from all this I really think we should comment out all the 0850 prefs by default.
If we do that then the description in the section title also needs some adjustments.
nothing more to add - as always, take it or leave it - this is good to go then. good job! Thanks!

I really think we should comment out all the 0850 prefs by default.

To clarify - I'd only comment out the 085x prefs. 086x are already commented out and 087x should definitely remain active because they "pollute" the C drive by creating a file in
C:\Users\<username>\AppData\Roaming\Microsoft\Windows\Recent\CustomDestinations
Idk if portable prevents that (it really should, imo) but I doubt that it does.
I think the ID in browser.taskbar.lastgroupid is what's used for the name of the file, if you want to check.

0807+0808 will stay separated - one is search bar, one is location bar (note 0808 says it requires 0807 to be turned on)
will do: shortened the test line for 0805
separation line for 0850+ nah, people need to learn to read, also it's not just browser.urlbar* so the wording would have to something else completely like possible relaxed settings section. It's all too messy. IF anything, I would gap it as per the suggestion in comment 4'ish way up there
commenting out all 0850+ items, idk (yes leave 087x active). I'm open to it (separation of app and os is really a hardened setting)... BUT

I am thinking of building in switches for easy flipping between branches - creating a baby bear and papa bear is easy, but mama bear is a bitch. We can revisit ALL of this in the branches, when we get to it .. in say 2 or 3 months. i.e we should look at the js in whole, rather than bits of it. And, by then, we should have a lot more users taking part, and we could take non-binding votes to get a community feedback (I say non-binding because we want to be consistent over the whole js - eg separation of app & os = commented out or not consistently and so on)
===
Random idea (no need to comment) .. just let the seed plant itself in your head
user_pref("app.update.enabled", false); // [MH]

L=lite M=medium H=hardened
so in the above, we only make the pref active in M & H versions
my thinking is that then we can just generate the new js's from code (simple to whip up a script) and do compares etc.

https://github.com/ghacksuserjs/ghacks-user.js/commit/37505e3461de28b053381340ac5e22312b3f8bee

one is search bar, one is location bar

ok, my bad, I read it as urlbar and location bar

separation line for 0850+ nah, people need to learn to read, also it's not just browser.urlbar*

what are you talking about? all the 085x prefs are browser.urlbar.*
I'm not talking about 0850**+**!

not really interested in adding a special line for 850s - they are already grouped as 850+letter - we currently don't do it elsewhere. Maybe we can revisit that later and be consistent across the js. And we've already said all we need to say in the section header, so "nah" :)


chore(workflows): include tag job in release

Triggering other workflows from within a workflow is
difficult/impossible, so couple these together.

@mason-org/triage
 🤖 This is an automated comment.    📖 Source code


Yeelight: use lazy initialization & add support for transitions

Lazy initialization is useful to allow adding bulbs not on during the time of startup.
Description: This moves the initialization routine to a separate function, which gets called when connectivity is needed. Allows adding bulbs not 'on' when homeassistant starts.
As a bonus it also adds support for transitions.
Checklist:
If user exposed functionality or configuration variables are added/changed:

[-] Documentation added/updated in home-assistant.github.io

If the code communicates with devices, web services, or third-party tools:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[-] New dependencies have been added to the REQUIREMENTS variable (example).
[-] New dependencies are only imported inside functions that use them (example).
[-] New dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[-] New files were added to .coveragerc.

looking forward to this patch

Scrapping this one, I'm working on replacement using another, more complete backend library which I hope will be able to get in as yeelight later on. The current state is https://github.com/rytilahti/home-assistant/commit/c70284e58f51742b25873dabe796f936428bfee8 and I've been using it for sometime locally, if someone wants to try it out.


Add clean script to TypeScript projects

In model v4, the entrypoint is a glob (dist/src/functions/*.js) meaning if you delete a ts file in src, you also need to delete the js file in dist otherwise the function will still show up. This can be confusing, so probably worth adding a clean script.
A few notes:

rimraf v5 works in Node.js v14+, but I feel like that's fine since Node 12 was EOL Oct 2020
There's not as much value when adding this to the old model, but I thought it was easier to keep them consistent
I'm pretty sure my test changes work, but obviously I wasn't able to do a full test run given the state of the tests

I'm pretty sure my test changes work, but obviously I wasn't able to do a full test run given the state of the tests


[WIP] logaggregator: bootstrap

Add logaggregator as a layer1 service.

Will also need adding to util/release/version_template.json

This will need adding to the updater's system apps (which will shortly move, see #1139) so that it gets updated.

Thanks for getting started on this, it's exactly what I needed to do next anyway :)

this should be ready to go. /cc @titanous, @bgentry

LGTM


chore: prepare release 2024-07-15

Summary
This PR has been created using the apic release script. Once merged, the clients will try to release their new version if their version has changed.
Version Changes

csharp: 7.0.0-beta.9 -> prerelease (e.g. 7.0.0-beta.10)
dart: 1.17.1 -> patch (e.g. 1.17.2)
go: 4.0.0-beta.23 -> prerelease (e.g. 4.0.0-beta.24)
java: 4.0.0-beta.40 -> prerelease (e.g. 4.0.0-beta.41)
javascript: 5.0.0-beta.11 -> prerelease (e.g. 5.0.0-beta.12)
kotlin: 3.0.0-beta.35 -> prerelease (e.g. 3.0.0-beta.36)
php: 4.0.0-beta.9 -> prerelease (e.g. 4.0.0-beta.10)
python: 4.0.0b24 -> patch (e.g. 4.0.0b25)
ruby: 3.0.0.beta.9 -> patch (e.g. 3.0.0.beta.10)
scala: 2.0.0-beta.9 -> prerelease (e.g. 2.0.0-beta.10)
swift: 9.0.0-beta.10 -> prerelease (e.g. 9.0.0-beta.11)

Skipped Commits
It doesn't mean these commits are being excluded from the release. It means they're not taken into account when the release process figured out the next version number, and updated the changelog.

Commits without language scope:

chore: release 2024-07-15 [skip ci]
chore: prepare release 2024-07-15 (#3378)
chore: keep guides for docs close to snippets (#3354)

Commits with unknown language scope:

fix(ci): run swift everytime, default success to false (#3379)
chore(deps): dependencies 2024-07-15 (#3356)
chore(ci): check the success of macos CI (#3353)

ah wait https://github.com/algolia/api-clients-automation/pull/3380


Add a common OCM setup troubleshooting guide

Signed-off-by: Mike Ng<EMAIL_ADDRESS>for https://github.com/open-cluster-management-io/OCM/issues/50

/lgtm


Unable to perform code signing on .dll files in av.libs

Dear maintainers of pyAV,
we are currently using pyAV in our audio transcription software 'aTrain'(GitHub). To properly distribute the app and provide trust in our code we need to perform code signing.
We use SignTool.exe from the .NET Framework-Tools to perform said code signing on Windows.
It is possible for us to sign every code and all .dll-files in our project except the .dll-files in pyAV's av.libs directory.
We tried to sign these .dll-file with a command similar to this one:
 signtool sign /a /fd SHA256 <PATH TO LIBRARY>\av.libs\avcodec-59-6fa00e8c4c19c8b4920003b2eb5d54f7.dll
Unfortunately, we get the following error:
Done Adding Additional Store
SignTool Error: SignedCode::Sign returned error: 0x800700C1
        For more information, please see https://aka.ms/badexeformat
SignTool Error: An error occurred while attempting to sign: venv\Lib\site-packages\av.libs\avcodec-59-6fa00e8c4c19c8b4920003b2eb5d54f7.dll

Number of errors: 1

This error only occurs when signing .dll-files in av.libs, all other .dll-files contained in other libraries of the project are possible to sign without errors.
My environment is as follows:

Operation System: Windows 11 Pro, Version 22H2
Python Version:  3.11.5
Version of pyAV: 10.0.0
singtool version as provided in the Windows 11 SDK (10.0.22621.0)

I already tried to find a solution by myself and encountered two possible reasons for the problem:

The dll-files are already signed and signtool therefore cannot sign them again. (see here)
I tried to remove possible existing code signatures even though I could not find signatures in the first place.
This did not fix the problem.

The dll-files have faulty PE headers (see here)
I could not find a proper solution to a problem with PE headers, since my knowledge of this matter is very limited.

I would be happy if you could direct me in a possible direction to resolve this issue.
Best regards,
Armin

We just provide the source code and the ffmpeg binaries we build in an unspecial way. You should talk to someone who specializes in Windows signing.


Bundling julia into the executable build dir

Sorry to bug you folks again, but would it be possible to give build_executable an option to bundle julia and all its dependencies into the executable for distributing to other machines that don't have julia installed?
For example, if I build my playground script on my host machine and try to transfer the build directory to a vm I get the following error.
$ ./playground
dyld: Library not loaded: @rpath/libjulia.dylib
  Referenced from: /Volumes/Playground.jl/deps/usr/build/./playground
  Reason: image not found
Trace/BPT trap: 5

From what I can tell the julia rpath is hard coded to the machine the executable was built on
otool -l playground
...
Load command 15
            cmd LC_RPATH
       cmdsize 96
           path /Users/rory/.playground/src/julia-0.4_2015-11-30/Contents/Resources/julia/lib/julia (offset 12)
...

I do not have a mac could you test if interchanging @executable_path in  https://github.com/dhoegh/BuildExecutable.jl/blob/aa248281d766c95161538b6b1f3980c55f58474a/src/BuildExecutable.jl#L179 with @loader_path fixes the problem?

So, that doesn't fix the problem and produces the same result cause sys.buildfile is always the full path on the host system cause of this line buildpath = abspath(dirname(Libdl.dlpath(debug ? "libjulia-debug" : "libjulia"))). I think the only way to get build_executable to do what I'm thinking is to have it take a flag that tells it to copy the julia build to the executable build directory and changes the buildpath accordingly. If that sounds reasonable I'll try and getting something working on the weekend when I have some time.

That sounds reasonable, as BuildExecutable produces a portable executable on windows but not on linux and mac. If you could figure out how to make the mac executable portable that would be greatly appreciated. As it would make the experience across platforms better and it could potentially also lead to a fix on to make the linux executable portable. I have tried briefly to look into the issue on linux, but did not manage.

Found a fix for OSX. Just needed an extra line to change the @rpath to point to @executable_path rather than the host julia install directory (sys.buildpath).

I will close this as i think this is fixed in #14. Please respond if this is not the case.


add latency line plots

Codecov Report

Merging #69 (50bb954) into master (d3a6324) will decrease coverage by 4.75%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master      #69      +/-   ##
==========================================
- Coverage   41.02%   36.27%   -4.76%
==========================================
  Files           6        6
  Lines         602      681      +79
==========================================
  Hits          247      247
- Misses        331      410      +79
  Partials       24       24

Impacted Files
Coverage Δ

cmd/dnspyre/plot.go
0.00% <0.00%> (ø)

cmd/dnspyre/report.go
55.55% <0.00%> (-0.32%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Run SQL autocomplete queries in a separate thread

Fixed error: DBConnectionException; must be caught or declared to be thrown

don't change the copyright banners.
also, i can't see the diff since the commit combines it with reformatting the code.


feat(template): Allow template output to use release name

helm template --output-dir command uses the chart name only when writing templates to disk. This causes collision in the path when  running the command for different releases of the same chart. This feature allows the release name to be used to avoid second invocation clobbering the previous output.
New path format: outputDir/<releaseName>/<chart>/templates
Closes #5664

Thanks for review @thomastaylor312


JSONWebToken header

Hi!
This project is awesome, so thank you for this contribution. But I do have a question - when turning my token String into JSONWebToken type I don't get header part in the object. Is this left out on purpose and if so, why? Otherwise it would be really great if we could access it the same way as the payload.

BUMP! Any updates on the issue?


Make the listeners return current scroll positions.

I propose implementing a system where the listeners for moved and stopped return the current scroll position.
Can probably by realised by changing Kinetic.prototype.stop and change the line this.settings.stopped.call(this); to this.settings.stopped.call(this,settings,{left: this.scrollLeft(),top: this.scrollTop());
According to the current documentation the stopped function also returns the settings, which it doesn't. It does work as documented in the moved function. This is fixed in the above mentioned change.
Should also be changed in Kinetic.prototype._move. Change  settings.moved.call(this, settings); to this.settings.moved.call(this,settings,{left: this.scrollLeft(),top: this.scrollTop());

this isn't really needed anymore since you can just access the current instance within the moved function ie:
moved: function() { console.log(this.scrollLeft()); }

good spot that the docs are outdated though

Alright, didn't know that I could access the instance from inside the callback. Good call, thanks!

no worries, i've updated the docs :smile:


Hybrid flow in IdentityServer4

Hi all,
I'm trying to implement hybrid flow in identity server 4. I have a custom scope, with custom claim, which I should get back from the UserInfoEndpoint. But:
when I make a request to response type "code id_token", I can get back the id_token in authentication, and my custom claim is in the id_token, and everything is working...
var options = new OpenIdConnectOptions { 	AuthenticationScheme = "oidc", 	SignInScheme = CookieAuthenticationDefaults.AuthenticationScheme, 	Authority = Configuration.GetSection("IdServerAuthenticationEndpoint").Value, 	RequireHttpsMetadata = true, 	ClientId = "TestClientMvc", 	ClientSecret = "secret", 	SaveTokens = true, 	ResponseType = "code id_token", 	GetClaimsFromUserInfoEndpoint = false, ...}
...but, when I try to get an access_token too (so, my custom claim should only return after a call to UserInfoEndpoint, and not represented in the id_token after authentication; and this is the way I should do it, if I understand well)...
... ResponseType = "code id_token token", ...
...I get the following error in authentication:

OpenIdConnectProtocolException: IDX10300: The hash claim: '...' in the id_token did not validate with against: '...', algorithm: 'RS256'.

Seems like the id_token becomes invalid somehow, when I request for access_token too. Did I misundarstanding/misconfiguring something, or is this an issue?

After some search (and debug): it doesn't seem to be an IdentityServer issue, it occurs on client side, and attached to this issue.
As for others in the future, use at least version 1.1.0 (if it will be released) of Microsoft.AspNetCore.Authentication.OpenIdConnect at the client (if you use it).


Updated publishing with Antora (with build script / readme)

What was done here?

docs_build.sh script that builds the documentation with all configured versions.

pipelines-docs have now a nicer url:
https://openshift.github.io/pipelines-docs/docs/0.8/index.html
instead of
https://openshift.github.io/pipelines-docs/docs/docs/0.8/index.html

README.md updated with Antora publishing steps

repo overhaul. Deleted a lot of unnecessary files.

Deleted asciidoc source files, as the current versions are sourced from different branches, to prevent clashes.

etc. I do not remember everything I fixed on the go :). TL;DR Publishing with Antora made somewhat easier.

Could you review @Preeticp @boczkowska, pretty please :)?

It's such neater! Looks great @pmacko1


#38 Working a Getting Started guide with IntelliJ IDEA 번역완료

오역은 천천히 잡아요

@uuuusssseeeerrrr 님 감사합니다!


Bugfixes

[x] Fix broken web example.
[x] Fix broken web2 example.
[x] Fix broken web4 example.
[x] Use getComputedStyle to calculate border width instead of hard-coding it in widget_selection.
[x] Fix missing $ (window.jQuery) error in widget_box.
[x] Fix occasionally broken destructuring initialization in widget_link.
[x] Formatting - remove trailing whitespace in widget.
[x] 	Only use floor or ceil when writing to the DOM in widget_selection.

cc @jasongrout @SylvainCorlay

Thanks a lot for this @afshin I think that the typescript transition is going to really improve the jupyter-js-widgets codebase.

Cheers! Yeah, I think you're right about TS :)

Thanks @afshin!


[BUG] T5-11B Inference Error

Describe the bug
I am trying to run T5-11B using deepspeed.init_inference. I am running on a g4dn.metal which as the following stats

The script i use:
# Coming from https://github.com/microsoft/DeepSpeed/issues/1332#issuecomment-1020617723

import os
import torch
import deepspeed
import transformers
import time
from deepspeed import module_inject
from transformers import pipeline
from transformers.models.t5.modeling_t5 import T5Block

# Get local gpu rank from torch.distributed/deepspeed launcher
local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))

print(
    "***************** Creating model in RANK ({0}) with WORLD_SIZE = {1} *****************"
    .format(local_rank,
            world_size))

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig

pipe = pipeline("text2text-generation", model="t5-11b")
print("model loaded on cpu")

assert pipe.model.config.num_heads % world_size == 0, f"{pipe.model.config.num_heads} % {world_size} = {pipe.model.config.num_heads % world_size} \nTo run the model parallel across the GPUs, the attention_heads require to be divisible by the world_size!This is because the attention computation is partitioned evenly among the parallel GPUs."

pipe.model = deepspeed.init_inference(
    pipe.model,
    mp_size=world_size,
    dtype=torch.float,
    injection_policy={T5Block: ('SelfAttention.o', 'EncDecAttention.o', 'DenseReluDense.wo')}
)

pipe.device = torch.device(f'cuda:{local_rank}')

print(
    f'({local_rank}) after deepspeed: allocated:{round(torch.cuda.memory_allocated()/1000/1000,2)}MB, cached:{round(torch.cuda.memory_allocated()/1000/1000,4)}MB'
)
print("check GPU memory usage")
time.sleep(10)

output = pipe("Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy")

if not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0:
    print(output)

How i execute the script python -m deepspeed.launcher.runner --num_gpus 8 t5_mp.py
The CLI output
[2022-02-15 09:13:38,583] [WARNING] [runner.py:155:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2022-02-15 09:13:38,830] [INFO] [runner.py:438:main] cmd = /usr/bin/python3.7 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNSwgNiwgN119 --master_addr=<IP_ADDRESS> --master_port=29500 t5_mp.py
[2022-02-15 09:13:39,586] [INFO] [launch.py:103:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]}
[2022-02-15 09:13:39,586] [INFO] [launch.py:110:main] nnodes=1, num_local_procs=8, node_rank=0
[2022-02-15 09:13:39,586] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]})
[2022-02-15 09:13:39,586] [INFO] [launch.py:123:main] dist_world_size=8
[2022-02-15 09:13:39,586] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
***************** Creating model in RANK (5) with WORLD_SIZE = 8 *****************
***************** Creating model in RANK (6) with WORLD_SIZE = 8 *****************
***************** Creating model in RANK (1) with WORLD_SIZE = 8 *****************
***************** Creating model in RANK (2) with WORLD_SIZE = 8 *****************
***************** Creating model in RANK (7) with WORLD_SIZE = 8 *****************
***************** Creating model in RANK (0) with WORLD_SIZE = 8 *****************
***************** Creating model in RANK (3) with WORLD_SIZE = 8 *****************
***************** Creating model in RANK (4) with WORLD_SIZE = 8 *****************
[2022-02-15 09:17:37,619] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 50394
[2022-02-15 09:17:37,623] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 50395
[2022-02-15 09:17:37,626] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 50396
[2022-02-15 09:17:37,626] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 50397
[2022-02-15 09:17:37,626] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 50398
[2022-02-15 09:17:37,627] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 50399
[2022-02-15 09:17:37,628] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 50400
[2022-02-15 09:17:37,628] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 50401
[2022-02-15 09:17:37,628] [ERROR] [launch.py:184:sigkill_handler] ['/usr/bin/python3.7', '-u', 't5_mp.py', '--local_rank=7'] exits with return code = -9

Expected behavior
It looks like the script exits before the init_inference is even called, but i don't know why. Is there a way to improve the error? technically it should be possible to load the model on CPU since the machine has 384GB Memory. Any idea why it is not working? is there something i am doing wrong?
ds_report output
Please run ds_report to give us details about your setup.
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
sparse_attn ............ [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [YES] ...... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/ubuntu/.local/lib/python3.7/site-packages/torch']
torch version .................... 1.10.2+cu113
torch cuda version ............... 11.3
nvcc version ..................... 11.3
deepspeed install path ........... ['/home/ubuntu/DeepSpeed/deepspeed']
deepspeed info ................... 0.6.0+5fe5b38, 5fe5b38, master
deepspeed wheel compiled w. ...... torch 1.10, cuda 11.3

cc @stas00 @RezaYazdaniAminabadi

Had to translate the table headers from German, so you have 384GB CPU memory and 128GB of GPU memory, 8x 16GB
So you're trying to load 40 x 8 = 320GB of CPU, and the way from_pretrained works it uses up 2x model size by default. so 40816 = 640 - can't fit and cgroups kills it - not sure why the system doesn't indicate that.
In order to use 1x cpu memory a low_cpu_mem_usage=True needs to be passed to from_pretrained. Perhaps for the sake of the experiment, hack the pipeline to use that. Actually I see in its API it has model_kwargs arg, so perhaps the fix here is:
pipe = pipeline("text2text-generation", model="t5-11b", low_cpu_mem_usage=True)

or did they mean to actually pass model_kwargs with a dict? have to scroll down a bit here to the API:
https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.pipeline
But really, in this situation clearly deepspeed-inference needs to provide a container like zero.Init which will split and instantiate the model directly on the gpus. Ideally it should re-use the zero.Init we are already using in from_pretrained here:
https://github.com/huggingface/transformers/blob/f45ac11fb32ba17e3e1fa831c30438ff9972a1ee/src/transformers/modeling_utils.py#L524-L531
BTW, does it work if you do t5-3b?

Please reopen if the issue still exists.

Hi @stas00 , I want to do t5-11b inference via deepspeed-inference and here is my scripts:

import torch
import os
from transformers import AutoTokenizer, AutoModelForCausalLM,AutoModelForSeq2SeqLM
import deepspeed
import torch.distributed as dist
from argparse import ArgumentParser

DEFAULT_DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

parser = ArgumentParser()
parser.add_argument("--local_rank", required=False, type=int, help="used by dist launchers")
parser.add_argument("--model_id", default= "t5-11b")

args = parser.parse_args()

local_rank = int(os.getenv("LOCAL_RANK", "0"))
world_size = int(os.getenv("WORLD_SIZE", "1"))
print(f'We have args now and local rank is {local_rank}, and world_size is {world_size}')

# Model Repository on huggingface.co
model_id = args.model_id

# load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
# tokenizer.pad_token = tokenizer.eos_token
# tokenizer.padding_side = 'left'

def print_model_rank0(model):
    if local_rank != 0:
        return
    for name,param in model.named_parameters():
        print(name, param.device)

model = AutoModelForSeq2SeqLM.from_pretrained(model_id, torch_dtype=torch.float16)
print_model_rank0(model)

# init deepspeed inference engine
ds_model = deepspeed.init_inference(
    model=model,      # Transformers models
    mp_size=world_size,        # Number of GPU
    dtype=torch.float16, # dtype of the weights (fp16)
    replace_method="auto", # Lets DS autmatically identify the layer to replace
    replace_with_kernel_inject=True, # replace the model with the kernel injector
)
# print(f"model is loaded on device {ds_model.module.device}")

print(f'We have at {local_rank}')
print(f'rank = dist.get_rank() {dist.get_rank()}')
print(f'Look at where model.device {model.device}')

print_model_rank0(ds_model)

# Test model
example = "My name is Philipp and I"
input_ids = tokenizer(example,return_tensors="pt").input_ids.to(DEFAULT_DEVICE)
logits = ds_model.generate(input_ids, do_sample=True, max_length=30)
print(f'local rank is {local_rank} and the generation is: {tokenizer.decode(logits[0].tolist())}')
#     'My name is Philipp and I live in Freiburg in Germany and I have a project called Cenapen. After three months in development already it is finally finished – and it is a Linux based device / operating system on an ARM Cortex A9 processor on a Raspberry Pi.\n\nAt the moment it offers the possibility to store data locally, it can retrieve data from a local, networked or web based Sqlite database (I’m writing this tutorial while I’'

And I have two questions here:

When I print the model device, I notice that all parameters belong to cuda:0, this is because TP? Like part of the weights are located at cuda 0
From here, I notice that at the second paragraph, it said :' It will automatically partition the model as necessary', so does that means that when we do smaller model like 't5-base', it actually wont do TP but just copy the whole model to another GPU since one model can fit into one GPU.

@ZeyiLiao, you probably want to tag @RezaYazdaniAminabadi on ds-inference questions as he is the author of this project and will be much better at answering questions than myself.

Hi @ZeyiLiao

Had to translate the table headers from German, so you have 384GB CPU memory and 128GB of GPU memory, 8x 16GB
So you're trying to load 40 x 8 = 320GB of CPU, and the way from_pretrained works it uses up 2x model size by default. so 40*8*16 = 640 - can't fit and cgroups kills it - not sure why the system doesn't indicate what did the killing.
In order to use 1x cpu memory a low_cpu_mem_usage=True needs to be passed to from_pretrained. Perhaps for the sake of the experiment, hack the pipeline to use that. Actually I see in its API it has model_kwargs arg, so perhaps the fix here is:
pipe = pipeline("text2text-generation", model="t5-11b", model_kwargs=dict(low_cpu_mem_usage=True))

But really, in this situation clearly deepspeed-inference needs to provide a container like zero.Init which will split and instantiate the model directly on the gpus. Ideally it should re-use the zero.Init we are already using in from_pretrained here:
https://github.com/huggingface/transformers/blob/f45ac11fb32ba17e3e1fa831c30438ff9972a1ee/src/transformers/modeling_utils.py#L524-L531
BTW, does it work if you do t5-3b?

For this issue, I am wondering if we can solve it by initializing this with meta tensors and then load the model onto GPU directly. @philschmid, did you try this with low_cpu_mem_usage and see if it resolves the issue as suggested by @stas00 ?
Thanks,
Reza

@RezaYazdaniAminabadi
Hi Reze, thanks for your reply and it solves my problem. I modify the code according your advice and add a new line ds_model._validate_model_kwargs({}).

import torch
import os

from transformers import AutoTokenizer, AutoModelForCausalLM,AutoModelForSeq2SeqLM
import deepspeed
import torch.distributed as dist
from argparse import ArgumentParser
from transformers.models.t5.modeling_t5 import T5Block

DEFAULT_DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

parser = ArgumentParser()
parser.add_argument("--local_rank", required=False, type=int, help="used by dist launchers")
parser.add_argument("--model_id", default= "t5-3b")

args = parser.parse_args()

local_rank = int(os.getenv("LOCAL_RANK", "0"))
world_size = int(os.getenv("WORLD_SIZE", "1"))
print(f'We have args now and local rank is {local_rank}, and world_size is {world_size}')

# Model Repository on huggingface.co
model_id = args.model_id

# load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
# tokenizer.pad_token = tokenizer.eos_token
# tokenizer.padding_side = 'left'

def print_model_rank0(model):
    if local_rank != 0:
        return
    for name,param in model.named_parameters():
        print(name, param.device)

def print_model_rank1(model):
    if local_rank != 1:
        return
    for name,param in model.named_parameters():
        print(name, param.device)

model = AutoModelForSeq2SeqLM.from_pretrained(model_id, torch_dtype=torch.float16)
print_model_rank0(model)

# t5 doesn't have supported kernel
# init deepspeed inference engine
ds_model = deepspeed.init_inference(
    model=model,      # Transformers models
    mp_size=world_size,        # Number of GPU
    dtype=torch.float16, # dtype of the weights (fp16)
    injection_policy={T5Block: ('SelfAttention.o', 'EncDecAttention.o', 'DenseReluDense.wo')}
)
# print(f"model is loaded on device {ds_model.module.device}")

print(f'We have at {local_rank}')
print(f'rank = dist.get_rank() {dist.get_rank()}')
print(f'Look at where model.device {model.device}')

print_model_rank0(ds_model)
print_model_rank1(ds_model)

# Test model
example = "My name is Philipp and I"
input_ids = tokenizer(example,return_tensors="pt").input_ids.to(DEFAULT_DEVICE)

ds_model._validate_model_kwargs({})

logits = ds_model.generate(input_ids, do_sample=True, max_length=30)
print(f'local rank is {local_rank} and the generation is: {tokenizer.decode(logits[0].tolist())}')
#     'My name is Philipp and I live in Freiburg in Germany and I have a project called Cenapen. After three months in development already it is finally finished – and it is a Linux based device / operating system on an ARM Cortex A9 processor on a Raspberry Pi.\n\nAt the moment it offers the possibility to store data locally, it can retrieve data from a local, networked or web based Sqlite database (I’m writing this tutorial while I’'

This is because I am modifying the inner code of generation and just have a function named generate(ds_model) and pass the model into this function. I followed what huggingface did, i.e. ds_model._validate_model_kwargs({}), but it said that
Traceback (most recen
t call last):
  File "test.py", line 73, in <module>
    ds_model._validate_model_kwargs({})
  File "/home/zeyi/miniconda3/envs/longtail/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1208, in __getattr__
    type(self).__name__, name))

AttributeError: 'InferenceEngine' object has no attribute '_validate_model_kwargs'
Traceback (most recent call last):
  File "test.py", line 73, in <module>
    ds_model._validate_model_kwargs({})
  File "/home/zeyi/miniconda3/envs/longtail/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1208, in __getattr__
    type(self).__name__, name))
AttributeError: 'InferenceEngine' object has no attribute '_validate_model_kwargs'
[2022-11-12 01:18:46,970] [INFO] [launch.py:286:sigkill_handler] Killing subprocess 87369
[2022-11-12 01:18:46,970] [INFO] [launch.py:286:sigkill_handler] Killing subprocess 87370
[2022-11-12 01:18:47,015] [ERROR] [launch.py:292:sigkill_handler] ['/home/zeyi/miniconda3/envs/longtail/bin/python', '-u', 'test.py', '--local_rank=1'] exits with return code = 1

How can I deal with it?
Also Can you help me about where can I find more details about injection_policy={T5Block: ('SelfAttention.o', 'EncDecAttention.o', 'DenseReluDense.wo')} since I am kind confused about what does this line mean. Just incase I may need to customize it if I encounter the same problem later.

Hi @ZeyiLiao, please open a new issue if you still need assistance. We are closing this since the original issue has been addressed.


Crash mid run of a small DB

Hi,
please see the attached crash log. the data is from a sample from teeth, searched against a small protein dataset, with GPTMD followed by the search. the search stage crashed.
Thanks!
David.
results.txt

are you willing to let us have a go?

sure. I will prepare a folder in the box

maybe the raw file is still loading?
i see fasta only

working on fix here: https://github.com/smith-chem-wisc/MetaMorpheus/pull/2453
Bug created by attempted addition of modification to a peptidewithsetmods that already had a modification at the same position

Thanks!
From: trishorts @.>
Sent: Friday, December 13, 2024 8:18 PM
To: smith-chem-wisc/MetaMorpheus @.>
Cc: David Morgenstern @.>; Author @.>
Subject: Re: [smith-chem-wisc/MetaMorpheus] Crash mid run of a small DB (Issue #2450)
Caution: External Sender. Do not click on links or open attachments unless you recognize the sender.
working on fix here: #2453https://github.com/smith-chem-wisc/MetaMorpheus/pull/2453
Bug created by attempted addition of modification to a peptidewithsetmods that already had a modification at the same position
—
Reply to this email directly, view it on GitHubhttps://github.com/smith-chem-wisc/MetaMorpheus/issues/2450#issuecomment-2542005160, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIQ5VIJYPOXSXAAGYWAEK5T2FMQF7AVCNFSM6AAAAABTQOUZGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBSGAYDKMJWGA.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>


Adicionar Documentação de rota de delete de ong

Describe the feature
Adicionar Documentação de rota de delete de ong

Adicionar a documentação no swagger na rota `DELETE /api/ongs/{ongId}` .
    O Guia para adicionar a documentacao do Swagger está no guia de contribuicao na raiz do projeto

  [Describe the new feature requested.]

Use Case

[Explain the use for this feature and how it might benefits the project.]

Implementation Details

[Provide any details or suggestions on how this feature could be implemented.]

Visual Concepts

[Include any visual representations or concepts if those are available and applicable.]

Additional Information

[Provide any additional information, such as context that might be relevant to the implementation of this specific feature.]

eu quero!!!


Add releaseid for release response

Required if you want to restore database data locally. Because modidstr+modversion or fileid is hard to use as a unique key. For example, here there is only modversion and fileid, and here only modidstr+modversion

👍

👍

this should be online now


Update Service Mesh Makefile targets

Add Gateway Channel selection using a new GW_CHANNEL variable. Defaults to "experimental" since most dev work will use that.
Also add a kind-servicemesh-install-cilium-fast target, which will install Cilium Servicemesh using the kind-image-fast and kind-install-cilium-fast flows, allowing local dev to update using make kind-image-fast without reinstalling the cluster when making code changes.

/test


Add piecewise linear codec, deprecate linear and mulitinear,  complete benchmark with real world datasets.

Ok PR is ready for review.
Here is a summary of the PR:

Piecewise linear codec added: basically it's a refactor and it fixes #1215
#1212 partially fixed that by saving 8 bytes.
Linear interpolation and multilinear deprecated. It's now only there for the reading part so that users can still read these fast field codecs
FOR codec added with feature unstable, it's not used in tantivy for the moment.
Bench also compression time/reading time and on real-world datasets.

Codecov Report

Merging #1217 (9a51e18) into main (02174d2) will decrease coverage by 0.09%.
The diff coverage is 78.52%.

@@            Coverage Diff             @@
##             main    #1217      +/-   ##
==========================================
- Coverage   94.05%   93.95%   -0.10%
==========================================
  Files         205      206       +1
  Lines       34514    34817     +303
==========================================
+ Hits        32463    32714     +251
- Misses       2051     2103      +52

Impacted Files
Coverage Δ

fastfield_codecs/src/lib.rs
96.87% <ø> (ø)

fastfield_codecs/src/main.rs
0.67% <0.00%> (-0.28%)
:arrow_down:

fastfield_codecs/src/multilinearinterpol_2.rs
95.51% <95.51%> (ø)

src/directory/file_watcher.rs
96.69% <0.00%> (-3.31%)
:arrow_down:

src/fastfield/reader.rs
94.06% <0.00%> (-0.85%)
:arrow_down:

src/directory/watch_event_router.rs
95.41% <0.00%> (-0.77%)
:arrow_down:

src/postings/postings_writer.rs
97.56% <0.00%> (-0.49%)
:arrow_down:

src/functional_test.rs
0.00% <0.00%> (ø)

src/positions/reader.rs
100.00% <0.00%> (ø)

src/reader/pool.rs
91.83% <0.00%> (+0.62%)
:arrow_up:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02174d2...9a51e18. Read the comment docs.

@PSeitz @fulmicoton : here is my take on what to do for next tantivy release and what to do after:
These remarks are valid for integers only (for floats, our codecs are useless):

piecewise linear interpolation should have the best compression ratio for monotonically increasing series
bitpacking should have the best compression ratio on random stuff
frame of reference (FOR) should have the best compression for series in between

What can we do for tantivy next release?

unplug linear interpolation and replace multilinear interpolation by new piecewise linear interpolation (the bug fix increases a little bit the compression ratio but nothing noteworthy)
update the benchmark with new datasets
I don't want to plug immediately the FOR stuff because I think we can do slightly better and I would also want to activate piecewise interpolation only on monotonically increasing series, I would introduce that later.

Future work:

finalize the FOR codecs, activate piecewise linear for only monotonically increasing series
sparse codecs
float codecs

Codecov Report

Merging #1217 (838a332) into main (447811c) will decrease coverage by 0.25%.
The diff coverage is 82.74%.

@@            Coverage Diff             @@
##             main    #1217      +/-   ##
==========================================
- Coverage   94.23%   93.98%   -0.26%
==========================================
  Files         232      234       +2
  Lines       40850    41416     +566
==========================================
+ Hits        38496    38923     +427
- Misses       2354     2493     +139

Impacted Files
Coverage Δ

fastfield_codecs/src/main.rs
0.54% <0.00%> (-0.40%)
:arrow_down:

fastfield_codecs/src/linearinterpol.rs
95.15% <92.85%> (-4.39%)
:arrow_down:

fastfield_codecs/src/frame_of_reference.rs
94.70% <94.70%> (ø)

src/fastfield/serializer/mod.rs
93.58% <96.66%> (+2.28%)
:arrow_up:

fastfield_codecs/src/piecewise_linear.rs
97.64% <97.64%> (ø)

fastfield_codecs/src/bitpacked.rs
100.00% <100.00%> (ø)

fastfield_codecs/src/lib.rs
96.92% <100.00%> (+0.04%)
:arrow_up:

fastfield_codecs/src/multilinearinterpol.rs
97.08% <100.00%> (-1.93%)
:arrow_down:

src/fastfield/mod.rs
92.93% <100.00%> (-0.02%)
:arrow_down:

src/fastfield/reader.rs
76.37% <100.00%> (-12.70%)
:arrow_down:

... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 447811c...838a332. Read the comment docs.

closing as obsolete. it was going in the right direction IMHO.


test: php 8.3

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (66c17bf) 99.65% compared to head (aefed73) 99.65%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##             master      #32   +/-   ##
=========================================
  Coverage     99.65%   99.65%
  Complexity       65       65
=========================================
  Files            15       15
  Lines           290      290
=========================================
  Hits            289      289
  Misses            1        1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Fix Promise dependency that breaks library in IE11

Bug Report
Current Behavior
Starting with v6.5.4, RxJS no longer works in IE11 without a Promise polyfill. The error is caused by the change in #5183.
https://github.com/ReactiveX/rxjs/blob/743c00bc520d7890e2782616f35503d57ae12441/src/internal/util/Immediate.ts#L2
Promise.resolve() is executed as soon as the library loads, preventing the use of RxJS without a Promise polyfill.
Reproduction
You can recreate this in a modern browser with:
<script>
  window.Promise = undefined;
</script>
<script<EMAIL_ADDRESS>
And get a TypeError:

Uncaught TypeError: can't access property "resolve", Promise is undefined

Expected behavior
I would expect RxJS to work in IE11, as long as I don't call functions that use Promise.
Environment

Runtime: Internet Explorer 11
RxJS version: 6.5.5 (last worked in 6.5.3)

Possible Solution
Switch back to using Promise.resolve()—and not a constant (RESOLVED)—in Immediate.setImmediate.

RxJS no longer works in IE11 without a Promise polyfill.

AFAICT, prior to that change you would still have needed a polyfill, as RxJS does not provide one. The change you referenced necessitates that Promise needs to be polyfilled before RxJS is loaded, but that is what applications should be doing anyway, so IMO this is not a bug.


Add test for nested grouping sets, rollup, cube

Add test cases because of spark supports nested grouping sets, roll-up, cube now.
This fixes #2781
Signed-off-by: Chong Gao<EMAIL_ADDRESS>

build

@revans2  @jlowe help to review

build


OpenUri not working with IP address

When the OpenUri target is an IP address, clicking the button only ever opens the URL : https://teams.microsoft.com/null

Hi @jparams,  we have a bug on this , this issue has been fixed and the fix is rolling in the internal rings. We have verified that internally and I will keep you updated once it will available publicly. Currently we don’t have any ETA to share.


Support Timezones

Support timezones properly.

Hey man, Is there a way I can get the plugin to recognise the local wordpress time settings. Its set to EDT and down here on the other side of the earth, thats a little inconvenient. ;-)
Cheers
Brett

I changed the three references of the New York time zone in the process file but its still displaying EDT. Is there anywhere else I to edit to get this function to change time zones to AEST.
Thanks in advance.

Hi @drinky,
I created a fix for this. Can you please test tag v1.1.0<PHONE_NUMBER>77460cf0a478f01423011cafca3b53).
Thanks!


Null Scans breaking metamorpheus during lowres CID search

probably fixed in #1841  yeah?

yes. maybe you could merge?


[FEATURE]: Scopes API

Feature hasn't been suggested before.

[X] I have verified this feature I'm about to request hasn't been suggested before.

Describe the enhancement you want to request
Scopes API
Background
Scopes or query scopes are the ability to add constraints to multiple queries without repeating yourself. Scopes provide the application with the ability to provide common SQL clauses, like JOINs and WHEREs, routinely.
These scopes are common in other ORM libraries, such as Ruby on Rails' ActiveRecord and Laravel's Eloquent. Currently of the TypeScript/JavaScript ORMs, only Sequelize provides scopes.
Scopes are able to be applied either globally across the entire codebase, working on a default level across every single query on the model/table unless otherwise specified, or conditionally on a per query basis. They provide an easy-to-use interface to implement common business logic by creating easily readable names.
For example, say instead of fully deleting a record from my users table when a user delete their account, I prefer to soft delete them. Perhaps I have a business logic reason I need to retain the data in the future. I create reated a column deleted_at to signify this. Throughout my application, I'd like to exclude by default exclude deleted users. Currently in Drizzle, this requires repeated logic like this:
import { users } from './drizzle/schema.ts'
import { isNull } from 'drizzle-orm'

const users = users.select().from(users).where(isNull('deletedAt'))

Or maybe I store another signifier for the country that the user is from. Maybe somewhere in my application, I'd like to only select U.S. users and repeat that query a few times:
import { users } from './drizzle/schema.ts'
import { db } from './drizzle/index.ts'
import { eq } from 'drizzle-orm'

const users = db.users.select().from(users).where(eq(users.country, 'US'))

But what if we could make that easier? In ActiveRecord, for example, the former look like:
Model File:
class User < ApplicationRecord
    # Model scope
    scope :active, -> { where(deleted_at: nil) }

    # Global scope
    default_scope { where(deleted_at: nil) }
end

Use in Application if Per-Model:
User.active.all

Use in Application if Globally Set:
User.all

Now if you want to remove the scope, you can either define an opposing scope if there are multiple default scopes and call it:
scope :including_deleted, -> { unscope(where: :deleted_at) }

Or if you there's only one global scope, you can unscope the entire model.
User.unscoped.all

Currently, to do similar in Drizzle ORM, you can define a function that you later call in each query. Here's a recent example I wrote for a project that uses Drizzle:
import { type SQL, and, sql } from 'drizzle-orm';
import type { PgColumn, PgTable } from 'drizzle-orm/pg-core';

interface SoftDelete {
    deletedAt: PgColumn<{
        name: string;
        tableName: string;
        dataType: 'date';
        columnType: 'PgTimestamp';
        data: Date | null;
        driverParam: string;
        notNull: false;
        hasDefault: false;
        isPrimaryKey: false;
        isAutoincrement: false;
        hasRuntimeDefault: false;
        enumValues: undefined;
        baseColumn: never;
        generated: undefined;
    }>;
}

export const excludeDeleted = <T extends PgTable>(table: T & SoftDelete): SQL<unknown> => {
    return sql`${table.deletedAt} IS NULL` as SQL<unknown>;
};

export const whereAndExcludeDeleted = <T extends PgTable>(
    table: T & SoftDelete,
    ...conditions: SQL<unknown>[]
): SQL<unknown> => {
    return and(excludeDeleted(table), ...conditions) as SQL<unknown>;
};

In Use:
const result = await db
        .select({ user: users, session: sessions })
        .from(sessions)
        .innerJoin(users, eq(sessions.userId, users.id))
        .where(whereAndExcludeDeleted(sessions, eq(sessions.id, sessionId)));

Proposal
To enhance Drizzle ORM's functionality and align it with best practices seen in other ORMs, we propose implementing a Scopes API. This API would allow for both global and per-query scopes, simplifying query logic and improving code maintainability.
Key Features:

Global Scopes: Automatically applied to all queries unless explicitly overridden. Ideal for common constraints like soft deletes.
Per-Query Scopes: Defined and applied on a per-query basis, allowing for flexible and reusable query logic.

Proposed API Syntax:
// Define a global scope in the schema
export const userScopes = scopes(users, {
  deleted_at: 'isNull(deletedAt)', global
});

// Automatically exclude deleted users
db.select.from(users);

// Define a one-off scope
export const userScopes = scopes(users, {
   active: 'isNull(deletedAt)'
});

// Apply scope per query
db.select.from(users).active;

The one important note here is that scopes should be able to be applied with other conditions. For example, if my applies a WHERE clause of deleted_at IS NULL to all queries, we need to be able to apply that and another condition on a per-query basis. The API needs to be intelligent enough to combine conditions without the needed use of and in the application code.
Benefits:

Maintainability: Reduces code duplication and potential for errors.
Readability: Provides a clear and concise way to express common query logic.
Flexibility: Allows for easy customization and extension of query behavior.

Next Steps:
I'd love feedback from the community on this proposal, particularly regarding the API design and potential use cases, and if the core maintainers would be open to this feature. If there's interest, I'd be happy to contribute to the implementation. Contributions and discussions are welcome to refine this feature further.

I think this would be better suited for RQB syntax. With SQL-like syntax, you can just create variables and use them as necessary in your queries, but can be a good addition to RQB which aims to add neat abstractions like this.


Fix app labels

Add labels metadata to templates so that applications are displayed properly in OCP console

@InfoSec812 Can we standardize on json for templates?

@mdanter Eric and Øystein are recommending YAML as the default. That said, I really have no preference. The problem is that once I extract and customize the template it is difficult to convert from one to the other. Suggestions for a way to easily convert everything to one or the other format would be greatly appreciated. The tools I have tried fail to be effective in either direction.

@InfoSec812 as to json vs yaml, I don't really care either. I'm fine with JSON going forward, but in this case, I don't think its a problem once my requested changes are made

Per https://bugzilla.redhat.com/show_bug.cgi?id=1503260, we need to update our automation to apply this label. I've opened https://github.com/redhat-cop/casl-ansible/issues/116 accordingly. Closing this PR