Support for dailymotion whitelabel embed url (/player/XXXXX.html)

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm requesting a site-specific feature
[X] I've verified that I'm running yt-dlp version 2023.07.06 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region
France
Example URLs

https://geo.dailymotion.com/player/x86gw.html?video=k46oCapRs4iikoz9DWy (click-to-embed on https://www.arretsurimages.net/chroniques/calmos/samuel-etienne-bienveillant-intergenerationnel-ou-boomer-sexiste)
https://geo.dailymotion.com/player/x86gw.html?video=k3gEvOgV9zsYNsz9KUa (also click-to-embed on https://www.arretsurimages.net/chroniques/calmos/samuel-etienne-bienveillant-intergenerationnel-ou-boomer-sexiste)
https://geo.dailymotion.com/player/xakln.html?video=x8mjju4&customConfig[customParams]=%2Ffr-fr%2Ftennis%2Fwimbledon-mens-singles%2Farticles-video (embedded on https://beinsports.com/fr-fr/tennis/wimbledon-mens-singles/articles-video/wimbledon-marketa-vondrousova-remporte-son-premier-tournoi-du-grand-chelem-2023-07-15)

Provide a description that is worded well enough to be understood
yt-dlp fails to download these videos, but the player seems to be nearly identical to the regular embedded player at https://geo.dailymotion.com/player.html, which is supported by yt-dlp. If i rewrite one of these URLs by hand from /player/XXXXX.html?video=YYYYYYY to /player.html?video=YYYYYYY then yt-dlp has no trouble downloading the video.
I believe the URLs are related to dailymotion's whitelabel player service. yt-dlp should support these urls.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['-vU', 'https://geo.dailymotion.com/player/x86gw.html?video=k46oCapRs4iikoz9DWy']
[debug] User config "/home/codl/.config/yt-dlp/config": ['--no-playlist', '--embed-subs', '--sub-langs', 'all,-live_chat,-rechat', '--add-metadata', '--format=bestvideo*[height<=1080]+bestvideo*+bestaudio/best+best[height<=1080]/best', '--video-multistreams', '--audio-multistreams']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>[b532a3481]
[debug] Python 3.11.3 (CPython x86_64 64bit) - Linux-6.1.37-1-lts-x86_64-with-glibc2.37 (OpenSSL 3.1.1 30 May 2023, glibc 2.37)
[debug] exe versions: ffmpeg 6.0 (setts), ffprobe 6.0, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.12.0, brotlicffi-<IP_ADDRESS>, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1855 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version<EMAIL_ADDRESS>Current version<EMAIL_ADDRESS>yt-dlp is up to date<EMAIL_ADDRESS>[generic] Extracting URL: https://geo.dailymotion.com/player/x86gw.html?video=k46oCapRs4iikoz9DWy
[generic] x86gw: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] x86gw: Extracting information
[debug] Looking for embeds
ERROR: Unsupported URL: https://geo.dailymotion.com/player/x86gw.html?video=k46oCapRs4iikoz9DWy
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1560, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1688, in __extract_info
    ie_result = ie.extract(url)
                ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 710, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/yt_dlp/extractor/generic.py", line 2568, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://geo.dailymotion.com/player/x86gw.html?video=k46oCapRs4iikoz9DWy

Based on your issue description (which I haven't verified), the solution should be as simple as modifying this regex


Codeunit 9856 "Permission Set Relation Impl." from internal to public

Describe the request
Change the access level of Codeunit 9856 "Permission Set Relation Impl." from internal to public
Additional context
Could you make the Codeunit 9856 "Permission Set Relation Impl." public instead of with access = internal?
I need to use the procedure RefreshPermissionSetTree to check which direct and nested permissions are assigned to a specific user.
I cannot replicate the functionality because some DotNet variables are used.
Thank you in advance for your support.

@Elio-Cometto-EOS Impl. codeunits contain the private methods that are never ever going to be made public.
You have a facade codeunit that is exposing the functionality.
Please go to the BCApps repo and open an issue to add RefreshPermissionSetTree to the public interface. If it gets approved you can contribute to the codebase and create a PR to extend the public facade.


Update README.md

Clarify required network mode in docker

I rewrited idea with other words in latest commit

I rewrited idea with other words in latest commit


Issue with react fiber

I'm trying to upgrade to react fiber and get this error with using the material-ui-upload:
Uncaught Error: Element ref was specified as a string (file-input) but no owner was set. You may have multiple copies of React loaded.

Haven't test it with fiber yet, will look on this soon.


Use protobuild to build protobufs

This PR uses protobuild tool to build protobufs, replacing the "go generate" command.
Signed-off-by: He Xiaoxi<EMAIL_ADDRESS>

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "use-protobuild" git@github.com:tossmilestone/swarmkit.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842353964368
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

Ok, this looks like a good start! I'm surprised that applying deepcopy everywhere just works! We might have to fix that later, but it seems to cause little harm here.
From a generation perspective, everything looks correct here. Please checkout a few comments I have on the paths for plugins/plugins.proto.

@stevvooe All problems addressed has been done, please help to review it again.

LGTM
I'll also be adding some API stability tools to protobuild that will benefit swarmkit!

To use the API stability tool of protobuild, I want to generate descriptor files of package api and protobuf/plugin, like api.pb.txt and plugin.pb.txt. What's your opinion? @stevvooe

@tossmilestone I think that is a great addition. Please do so in a follow up PR.

@stevvooe OK


Bug report: "m365 spo listitem set" issues with Number and Single Line of Text

Description
There are two closely related issues. One is setting a Single Line of Text with a string which contains something which looks like an integer. The other is simply setting a Number.

If you try to set the value of a Single Line of Text with a string as such "example", it will work. If you used "123example", it will work. If you use "123", it will say:
Error: Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details.

If you try to set the value of Number, it doesn't matter if you put the value in quotes or not, you get
Error: Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details.

Steps to reproduce
Create a list with a Single Line of Text called Name and a Number called Total.
C:>m365 spo listitem set -u "https://mytenant.sharepoint.com/sites/my-site" --listTitle "Test" --id 1 --Name "123"
Error: Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details.
C:>m365 spo listitem set -u "https://mytenant.sharepoint.com/sites/my-site" --listTitle "Test" --id 1 --Total "123"
Error: Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details.
C:>m365 spo listitem set -u "https://mytenant.sharepoint.com/sites/my-site" --listTitle "Test" --id 1 --Total 123
Error: Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details.
Expected results
Should be able to set a Single Line of Text to a string value "123". Should be able to set a Number to a value 123.
Actual results
Error: Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details.
Diagnostics
In both cases, the request contains  a FieldValue which has no quotes
"data": {
"formValues": [
{
"FieldName": "Name",
"FieldValue": 123
}
]
}
"data": {
"formValues": [
{
"FieldName": "Total",
"FieldValue": 123
}
]
}
and the response
"status": 400,
"statusText": "Bad Request",
"error": {
"odata.error": {
"code": "-1, Microsoft.Data.OData.ODataException",
"message": {
"lang": "en-US",
"value": "Cannot convert a primitive value to the expected type 'Edm.String'. See the inner exception for more details."
}
}
CLI for Microsoft 365 version
v5.3.0
nodejs version
v14.19.0
Operating system (environment)
Windows
Shell
cmd
cli doctor
{
"os": {
"platform": "win32",
"version": "Windows 10 Enterprise",
"release": "10.0.19044"
},
"cliVersion": "5.3.0",
"nodeVersion": "v14.19.0",
"cliAadAppId": "31359c7f-bd7e-475c-86db-fdb8c937548e",
"cliAadAppTenant": "common",
"authMode": "DeviceCode",
"cliEnvironment": "",
"cliConfig": {},
"roles": [],
"scopes": [
"AllSites.FullControl",
"AppCatalog.ReadWrite.All",
"Directory.AccessAsUser.All",
"Directory.ReadWrite.All",
"Group.ReadWrite.All",
"IdentityProvider.ReadWrite.All",
"Mail.Send",
"Reports.Read.All",
"TermStore.ReadWrite.All",
"User.Invite.All",
"User.Read.All",
"profile",
"openid",
"email",
"AllSites.FullControl",
"AppCatalog.ReadWrite.All",
"Directory.AccessAsUser.All",
"Directory.ReadWrite.All",
"Group.ReadWrite.All",
"IdentityProvider.ReadWrite.All",
"Mail.Send",
"Reports.Read.All",
"TermStore.ReadWrite.All",
"User.Invite.All",
"User.Read.All"
]
}
Additional Info
No response

Hi @preavy, I was able to reproduce your problem so I did a little digging to see how we can solve this for you.
First of all, with my testing I got the following results. On my test environment I have a list called Demo with 2 columns. Column SLT is a single line of text and column Number is, well a number column.

As you can see, the results I got are quite similar towards your findings.
But not to worry. I found a solution for you, but this requires an update towards the CLI. So for this, I will open a PR and as soon as this is approved you will be able to modify number columns and text columns with only numbers. 😄

Thanks for reporting this @preavy!

Thanks for this. My use case was copying items from one SharePoint list to another. I was aiming to use m365 cli alone to do this but in the meantime I was able to work around this issue by mixing in Set-PnPListItem from PnP.PowerShell. I appreciate you looking into the issue.


i have this issue but i'm not sure how to debug or what it means. how to simulate the issue?

i have this issue but i'm not sure how to debug or what it means. how to simulate the issue / problem / error?
what does it mean?
server.go:1894: [Error] HERTZ: HTTP2 write response body error when serving <IP_ADDRESS>:26952: http: request method or response status code does not allow body

The status code is 204 but there is also a body?

@rogerogers how to check which status codes have body and which doesnt? i'm writing a customized reverse proxy so anything returned with "body" will be used in setbody

For this specific problem, you can check out the method https://github.com/hertz-contrib/http2/blob/e0eecbe2edd15b04264f8b7ac345e03b021e5193/http2.go#L309. For further information, you need to Google.

@rogerogers how to check which status codes have body and which doesnt? i'm writing a customized reverse proxy so anything returned with "body" will be used in setbody


Usage with MVC bundles

Hello,
First thanks for this great project, it has helped me a lot. But now I ran into a problem I don't know how to solve...
I'm trying to use EmbeddedResourceVirtualPathProvider as VirtualPathProvider for MVC bundles..
However I have a problem that my bundles are not refreshing when I edit files.
Here's my usage:
BundleTable.VirtualPathProvider = new Vpp(BundleTable.VirtualPathProvider)
{
{typeof (BundleConfig).Assembly, @""},
};
It does not work if file is embeded or not... Not sure on how to solve this issue.
Thanks for any help !

I'm not sure whats wrong. If you update the example project to use Mvc bundles, and push that to a branch, I could take a look.

This is probably related to https://github.com/mcintyre321/EmbeddedResourceVirtualPathProvider/issues/11


[Suggestion] Compatibility with Valkyrien Skies

Minecraft Version

[ ] 1.12.2 (End of support)
[ ] 1.16.5 (End of support)
[ ] 1.18.2
[ ] 1.19.2
[X] 1.20/1.20.1

What happened?
I have noticed that you can seat maids on seats from create - but that is not possible if the seat is placed on a valkyrien skies ship.
It's also not possible to place maid chairs on valkyrien skies ships.
I just wanna ride around with my waifu :) (and sorry if this is the wrong place to ask for such thing)
Relevant log output
No response
Contact Details
No response

Download
https://www.dropbox.com/scl/fi/ku9a1wblqyb84rb8ekase/fix.zip?rlkey=8763vim31xfgywjgy217yb8lh&st=gbp0kafn&dl=1
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."


Better Menu (Splash Screen)

Assets:

[x] Menu Frame (good border, little transparency (alpha = 70 - 80%), dark background)

Files in 'Asset' folder of the drive.


REST API And Database

Im looking for people can show me rest api url for curl, not source code also if developer can adjust code to create database for mysql because i want help do php administrator and user UI with by connecting API to mysql database. Im sorry i request this because i still new in python also doesn't anything about golang. I only expert in PHP, CSS, HTML and MYSQL and little JS.

This feels very off topic to Boring Proxy. Please try to find help in other online spaces dedicated to skill sharing, like StackOverflow. Good luck!

This feels very off topic to Boring Proxy. Please try to find help in other online spaces dedicated to skill sharing, like StackOverflow. Good luck!

I don't think you understand my question. I not mention boringproxy on there not meaning its off topic. Why should i go stackoverflow? What i request is REST api (not source) from this project. Because i want help/support this project by creating full admin and client web UI. Its like user can register, reset password, notification, telegram bot and more based of this API. About database it will convert database json to mysql meaning its useful to handle big data.

Hey @vqoley, I understand your request better now. Unfortunately, while boringproxy does have a programmable API, it's not currently encouraged for other developers to use it, since it is unstable and may change a lot. Also, there's no documentation for it.
I think boringproxy probably is too immature of a project for what you're trying to accomplish. I'd recommend trying to find something that's past version 1.0 with a stable API.
Let me know if you have any more questions.


Error in img2img

/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py:749: FutureWarning: torch_dtype is deprecated and will be removed in version 0.25.0.
deprecate("torch_dtype", "0.25.0", "")
/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py:752: FutureWarning: torch_device is deprecated and will be removed in version 0.25.0.
deprecate("torch_device", "0.25.0", "")
Traceback (most recent call last):
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1431, in process_api
result = await self.call_function(
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1103, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "/home/user/stable-diffusion-webui/extensions/sd-webui-lcm/scripts/main.py", line 172, in generate_i2i
result = pipe(
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/user/stable-diffusion-webui/extensions/sd-webui-lcm/lcm/lcm_i2i_pipeline.py", line 305, in call
self.scheduler.set_timesteps(strength, num_inference_steps, original_inference_steps)
File "/home/user/stable-diffusion-webui/venv/lib/python3.10/site-packages/diffusers/schedulers/scheduling_lcm.py", line 382, in set_timesteps
timesteps = lcm_origin_timesteps[::-skipping_step][:num_inference_steps]
TypeError: slice indices must be integers or None or have an index method

same error here and in LCM vid2vid

same here

there is a problem of function overlap with diffusers/src/diffusers/schedulers
/scheduling_lcm.py

@Ehplodor I do not think so. If you setup denoising strange to 1 there is problem with other part of code and so on... this project is abandoned liki for ComfyUI

@ato-zen yes, the parameters of set_timesteps aren't aligned between each version of the function (the one in diffusers here has evolved in contrasts to the one implemented into lcm i2i pipeline here)

set_timestep is called here on line 305 of lcm i2i pipeline but it calls diffusers version where params are different

So lcm's denoise strength is actually sent into num_inference_steps from diffuser's version of set_timestep().

hence the problem with non integer indices if denoise strength is set to anything different than 1.0

@ato-zen ok i understand why you say it's been abandoned. Code is obsolete and would need multiple evolutions. TY for pointing that out. Do you know of an alternative for A1111 webui ?

LCM checkpoints XL and LCM 1.5 SD is blazing fast for upscaling with tile
https://postimg.cc/1fMPcXcf


How to recover index of surface.

TL;DR: How do I recover the corresponding backface irradiance to compare to my reference cell.
Hello,
I am trying to recover the backface irradiances with the new API.
Probably i am just missing something, but I can't find a way to quickly recover the indexes.
I am doing this:
def individual_report(pvarray:OrderedPVArray, row:int=0):
    back_surface_list = pvarray.ts_pvrows[row].back.list_segments
    return pd.DataFrame({f'qinc_{s.index}':s.get_param_weighted('qinc') for s in back_surface_list})

but the s.index is always none.
If I index using all_ts_surfaces I get way too much segments. I know that al surfaces are listed, including shaded elements.
Any tips?

Ok,
There are TsSegment and TsSurface, but the indexes I plotted only show surface, but not segments.
This is pretty confusing btw...

Hey @tcapelle , thanks a lot for flagging this! Sorry I haven't been very active on pvfactors lately, I'm a bit buried at work

It does seem like the segments' index attribute is never set, good catch! It should be an easy fix, just changing this line to:

segment = TsSegment(segment_coords, illum, shaded,
                                n_vector=n_vector, index=i)

I'll try to do it this week, but feel free to open a PR if you can!

In the meantime for your situation, you could just use the segment's index in the list of segments for now (it's exactly what's going to be used after the fix):

def individual_report(pvarray:OrderedPVArray, row:int=0):
    back_surface_list = pvarray.ts_pvrows[row].back.list_segments
    return pd.DataFrame({f'qinc_{idx}':s.get_param_weighted('qinc') for idx, s in enumerate(back_surface_list)})

I'd be more than happy to help here; could you just clarify what you find confusing and what solution you think could help?

Ok,
There are TsSegment and TsSurface, but the indexes I plotted only show surface, but not segments.
This is pretty confusing btw...

Thanks a lot for opening this issue!


fix: add types to test configuration

This adds types to our test configuration (added in https://github.com/safe-global/safe-client-gateway-nest/commit/2a2168f37dd4b2c79ddd6fe92ad4942c4c870f0e), updating incorrect values accordingly.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.07%) to 92.035%

Files with Coverage Reduction
New Missed Lines
%

src/routes/transactions/mappers/common/transaction-info.mapper.ts
1
77.59%

Totals

Change from base Build<PHONE_NUMBER>:
-0.07%

Covered Lines:
4147

Relevant Lines:
4394

💛  - Coveralls


dalle use a separate key

Sometimes we use different apikeys, so suggestted to separate them.

Thank you, but that’s out of scope and extremely niche. I’m not going to add Workflow Environment Variables or configurations for overriding API keys, those are foot guns waiting to fire on inexperienced users.


Missing license

Hi,
How is this crate licensed? 🙂
Edit: I now found the license field in Cargo.toml. A separate file would make this more obvious.

Hah, I'll admit for a moment I was wondering what you're talking about, of course this crate has a license - but then I saw there's no LICENSE file; I'll add one. Currently, the only (and canonical) place listing the license is the governor crate's Cargo.toml file - it's MIT.


GO:0036038 update

Hi, I would like to rename the GO:0036038 tectonic complex into 'tectonic-like complex'. A new paper published a similar complex which they named
B9 complex (PubMed=22179047) and characterizes its function
Many thanks
Sylvain
Here are the modifications I propose:
GO:0036038 tectonic-like complex
A protein complex that is located at the ciliary transition zone and consists of tectonic proteins (TCTN1, TCTN2), B9 domain-containing proteins (B9D1, B9D2, MKS1) and other proteins. Acts as a barrier that prevents diffusion of transmembrane proteins between the cilia and plasma membranes.
PubMed=22179047;
Synonym=B9 complex
Reported by: sylvainpoux
Original Ticket: geneontology/ontology-requests/9103

Hi,
I was annotating a paper on C. elegans ciliary transition zone proteins and would like to make some annotations to the appropriate protein complexes that reside and function there.
First, are we still adding new protein complex terms to the CC ontology?
If yes, then from some reading, it looks like there are two main protein complexes, referred to in recent papers as 'modules', that would be appropriate for annotation:  the MKS/JBTS module and the NPHP module.
I believe the MKS/JBTS module corresponds to GO:0036038 and so I'm wondering if we should re-name this term, since it looks like the MKS nomenclature is what is currently used in the literature.
The NPHP module could be broken down into two separate modules, the NPHP1-4-8 module and the NPHP5-6 module.  Both would be new CC terms, also children of GO:0035869 ciliary transition zone.
If we make these changes, I don't know if we want to use the 'module' nomenclature or just refer to these as the MKS/JBTS complex and the NPHP complex.  I think the reason 'module' is used in the literature is that:
"...proteins in the MKS–JBTS module (green) mostly interact with other proteins within the same module and only rarely with those in the NPHP module (each line designates a reported protein–protein interaction)."
In other words, the modules are not wholly distinct complexes, but are believed to have some association between them, albeit one that is considerably less strong than the association within the individual modules.
See:   http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3373398
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3383065/
Please let me know if you'd like more information.
Thanks,
--Kimberly

Hi Kimberly,
I have re-opened this ticket so it doesn't fall through the cracks, but I'm afraid I won't have a chance to look into it too soon. Please let me know if it is very urgent.
Thanks, and a good weekend,
Paola

Thanks, Paola.
I can annotate the gene products to the parent term, 'ciliary transition zone', in the meantime, so this is not urgent.
If you need any more info, let me know.
Enjoy your weekend, too.
--Kimberly

@vanaukenk @krchristie @JohnvanDam
Hi, please see Kimberly’s comment dated Nov 6, 2015.
Questions:

Are the MKS–JBTS and NPHP modules/protein complexes stable complexes, or are they transient? We tend to consider transient complexes outside the scope of GO.
(See http://wiki.geneontology.org/index.php/Guidelines_on_'protein_complex'_terms)

If they are stable complexes, would they be useful for annotation?

Is MKS–JBTS the same as GO:0036038 TCTN-B9D complex? If yes, is the former the preferred nomenclature in cilia literature?

If this applies: please suggest definitions for MKS–JBTS and/or NPHP modules/protein complexes, and functions that they are capable of if known, and processes they are involved in if known.

Thanks,
Paola

Hi Paola,
I'm under the impression that the MKS and NPHP modules are stable complexes involved in the structure of the transition zone that separates the ciliary plasm from the cytoplasm. Here is an interesting paper discussing them.
-Karen
Barker AR, Renzaglia KS, Fry K, Dawe HR. Bioinformatic analysis of ciliary
transition zone proteins reveals insights into the evolution of ciliopathy
networks. BMC Genomics. 2014 Jun 26;15:531. doi: 10.1186/1471-2164-15-531. PubMed
PMID: 24969356; PubMed Central PMCID: PMC4092220.

Thanks Karen. The reference you suggest
http://bmcgenomics.biomedcentral.com/articles/10.1186/1471-2164-15-531
is very informative, but I feel that we still don’t have enough elements to define these complexes in GO, and I’m not sure it’d be useful to have these terms in until more is known about their function (we don't have facts to link them to precise processes either). These uncertainties are summarised in the conclusions:
“Given the near-ubiquity of TMEM67, CC2D2A, B9D1, B9D2, AHI1, a single TCTN family member, and possibly MKS1 across ciliated eukaryotes [see fig 2], we suggest a dynamic complex built around these proteins and varying in a spatio-temporal and/or organism/tissue-specific manner. In concordance with current ideas, formation of this complex is likely to be key in the proper formation and functioning of ciliary Y-links.”
Therefore, @vanaukenk, I’d rather not add ‘MKS/JBTS module’ and ‘NPHP module’ to GO, but I could add them (and ‘MKS module') as narrow or related synonyms of ‘ciliary transition zone’, adding the papers you and Karen quoted as dbxref. Let me know if you think this would be helpful.
Thanks,
Paola

Hi @paolaroncaglia,
There are a couple of new papers that have come out this year about the role of these different modules in transition zone assembly, so if it's alright, I'd like to just take a look at those papers to come back up to speed on this issue.
In the literature, people tend to talk about the MKS and NPHP complexes as separate functional units based on several criteria, e.g. genetic interactions, physical interactions, and localization dependencies, but I take your point that we want good definitions of these modules, from both a compositional and functional point of view, to consider adding them to GO.
Thanks,
--Kimberly

Are the MKS–JBTS and NPHP modules/protein complexes stable complexes, or are they transient? We tend to consider transient complexes outside the scope of GO.
(See http://wiki.geneontology.org/index.php/Guidelines_on_'protein_complex'_terms)

They are considered stable complexes, or at least stably co-localise to the transition zone and influence each others localisation. I have recently been involved in a large study to identify a novel component of the MKS module and the very nice thing was that my collaborator did some very systematic experiments. The NPHP and MKS modules are very hierarchical in that some components can be removed while module integrity is maintained causing only minor phentypes, while others will disrupt the whole system and cause severe phenotypes including prenatal death. We are only now finding out how these proteins are related to actual structures (before they were only found to be "required" for Y-shaped linker formation, but we now finally find evidence that they are actual "part of" that structure, this is still thin though). Literature wise, these modules are becoming increasingly important in ciliary biology because of their strong relation to human genetic disease.

If they are stable complexes, would they be useful for annotation?
They are useful for annotation in that there is a lot of evidence that suggest that these are functional units and are often found mutated in human genetic disease (ciliopathies). They perform functions in ciliary protein transport, protein barrier function, and cellular signalling (although that could likely be indirect)

Is MKS–JBTS the same as GO:0036038 TCTN-B9D complex? If yes, is the former the preferred nomenclature in cilia literature?

The TCTN-BD9 complex seems to be (according to genes assigned) a more "broad" definition of the MKS module as how we used it. The NPHP module is separate and also not part of the TCTN-B9D. I think "MKS module" or "MKS complex" is more common in literature, but have no numbers.

If this applies: please suggest definitions for MKS–JBTS and/or NPHP modules/protein complexes, and functions that they are capable of if known, and processes they are involved in if known.

I will get back to you this afternoon with specific proposal for definitions of these complexes including references, parent and child terms etc.
In the meantime this is the paper I have been involved in:
Lambacher NJ, Bruel A-L, van Dam TJP, Szymańska K, Slaats GG, Kuhns S, et al. TMEM107 recruits ciliopathy proteins to subdomains of the ciliary transition zone and causes Joubert syndrome. Nat. Cell Biol. Nature Publishing Group; 2015;18:122–31. http://dx.doi.org/10.1038/ncb3273

So many distractions... I'll get you the proposed definitions tomorrow. Sorry for this.

"TCTN-B9D" -> no hits in pubmed
"MKS-complex" -> 4 hits
"mks-module" -> no hits
"mks-jbts" -> no hits
"mks module" -> 7 hits
“nphp module” -> 6 hits
“nphp-module” -> 0 hits
“nphp-complex” -> 0 hits
“nphp complex” -> 21 hits
The TCTN-B9D complex comprises many elements of the so called MKS module, so we can consider these to be essentially the same, which some exceptions. The distinction between NPHP module and MKS module comes from the fact that the proteins of either module can be deleted/knocked down without loss of structure, however when both modules are disrupted (e.g. by double gene knockout like NPHP4 and MKS1) the transition zone becomes disorganised and y-shaped linkers disappear or are misaligned or dislodged from membrane and axoneme (genetic interaction). Between most module components physical interactions have been measured, but the question remains if this is transient in nature. The localisation of many MKS/NPHP module components are highly restricted to the TZ, but have only recently been shown to be “fixed” within the structure for one or two components, suggesting to some extent a non-transient nature of the modules. It has been shown that a new component of the MKS module (TMEM107) is fixed at the y-shaped attachments to the membrane suggesting that the MKS module may form the anchor of the Y-shape to the membrane. Since many MKS proteins have transmembrane regions this may very well be. The NPHP module is smaller in size and to my knowledge none of the components contain TM regions.
Currently MKS module seems to be used more frequently, but I think this would be counter the normal nomenclature in GO (i.e. complex). I suggest to drop the TCTN-B9D name since it is not used in literature. CEP290 is currently annotated as being part of the TCTN-B9D complex. The CEP290 protein is generally not considered “part of” the MKS module although it is tightly associated to it functionally (PMID: 26982032).  Some other TZ components that are not considered to be part of this complex are also assigned to this term like AHI1.
Overall I think the TCTN-B9D term needs an overhaul. Below is my suggested name and definition including new references.
Name: MKS module
Ontology: cellular_component
synonyms: MKS complex, tectonic complex, tectonic-like complex
Definition: A protein complex that is located at the ciliary transition zone and consists of several proteins some of which are membrane bound. Acts as an organiser of transition zone inner structure, specifically the Y-shaped linkers, in conjunction with the NPHP complex. The complex also acts as part of the selective barrier that prevents diffusion of proteins between the ciliary cytoplasm and cellular cytoplasm as well as between the ciliary membrane and plasma membrane.
Source (old): PMID:22179047, PMID:21725307
Source (new): PMID:26595381, PMID:21422230, PMID:25869670, PMID:26982032, PMID:21565611
Comment: Although there is some evidence, it is unclear whether the MKS- and NPHP complexes are truly “part of” the y-shaped linkers or are just responsible for aligning and attaching the y-shaped linkers to the membrane and axoneme.
relationships:
is_a protein complex
part_of ciliary transition zone
is_a ciliary part
I'll draft a suggested definition of the NPHP module next. I suggest to not define the "newer" CEP290 module in GO right now, since this term is, I believe, not widespread.
Apologies for the long read. I'll try to keep it shorter next time.

Found this while searching for references. I am adding it to this thread as a side note:
Williams CL, Li C, Kida K, Inglis PN, Mohan S, Semenec L, et al. MKS and NPHP modules cooperate to establish basal body/transition zone membrane associations and ciliary gate function during ciliogenesis. J. Cell Biol. 2011;192:1023–41.

Hi @JohnvanDam ,
I edited the existing term GO:0036038 TCTN-B9D complex based on your input. The new stanza is below. (Please note, we’d rather have ‘complex’ in the primary name for consistency with other protein complex terms; I made the ‘module’ wording an exact synonym.)
When you have a chance, could you please suggest definition etc. for the NPHP module/complex, as you kindly offered to? :-)
Thanks.
[Term]
id: GO:0036038
name: MKS complex
namespace: cellular_component
def: "A protein complex that is located at the ciliary transition zone and consists of several proteins some of which are membrane bound. Acts as an organiser of transition zone inner structure, specifically the Y-shaped links, in conjunction with the NPHP complex. The MKS complex also acts as part of the selective barrier that prevents diffusion of proteins between the ciliary cytoplasm and cellular cytoplasm as well as between the ciliary membrane and plasma membrane." [GOC:cilia, GOC:sp, PMID:21725307, PMID:22179047, PMID:26595381, PMID:21422230, PMID:25869670, PMID:26982032, PMID:21565611]
comment: Although there is some evidence, it’s still unclear if the MKS and NPHP complexes are constituents parts of the Y-shaped links or are simply responsible for aligning and attaching the Y-shaped links to the membrane and axoneme.
synonym: "B9 complex" NARROW []
synonym: "MKS module" EXACT []
synonym: "TCTN-B9D complex" BROAD []
synonym: "tectonic complex" NARROW []
synonym: "tectonic-like complex" NARROW []
is_a: GO:0043234 ! protein complex
is_a: GO:0044441 {is_inferred="true"} ! ciliary part
relationship: part_of GO:0035869 ! ciliary transition zone

Note for self:
Once the NPHP term is added, I can close this ticket.

Name: NPHP complex
namespace: cellular_component
def: "A protein complex that is located at the ciliary transition zone and consists of the NPHP4 and NPHP1 proteins. Acts as an organiser of the transition zone inner structure, specifically the Y-shaped links, in conjunction with the MKS complex. Mutation or deletion of individual NPHP complex components cause disruption in ciliary protein trafficking, but cause large structural defects when MKS complex components are also mutated. The NPHP complex is required for correct functioning of the WNT and Hippo signaling pathways." [PMID:21422230,PMID:25150219,PMID:21555462,PMID:21498478,PMID:18337471]
comment: Although there is some evidence, it’s still unclear if the MKS and NPHP complexes are constituents parts of the Y-shaped links or are simply responsible for aligning and attaching the Y-shaped links to the membrane and axoneme.
synonym: "NPHP module"
is_a: GO:0043234 ! protein complex
is_a: GO:0044441 {is_inferred="true"} ! ciliary part
relationship: part_of GO:0035869 ! ciliary transition zone

Thanks @JohnvanDam !
I added GO:1990957 ‘NPHP complex’ as below (I modified your suggested definition slightly so we don’t quote mutagenesis experiments directly, but we say what the results of those experiments indicate; and I added links to the relevant biological processes). Closing now. Here’s the full stanza for reference:
[Term]
id: GO:1990957
name: NPHP complex
namespace: cellular_component
def: "A protein complex that is located at the ciliary transition zone and consists of the NPHP4 and NPHP1 proteins. It acts as an organiser of the transition zone inner structure, specifically the Y-shaped links, in conjunction with the MKS complex. It is involved in ciliary protein trafficking and is required for correct functioning of the WNT and Hippo signaling pathways." [GOC:cilia, PMID:18337471, PMID:21422230, PMID:21498478, PMID:21555462, PMID:25150219]
comment: Although there is some evidence, it's still unclear if the MKS and NPHP complexes are constituents parts of the ciliary Y-shaped links or are simply responsible for aligning and attaching the Y-shaped links to the cilium membrane and axoneme.
synonym: "NPHP module" EXACT []
is_a: GO:0043234 ! protein complex
is_a: GO:0044441 {is_inferred="true"} ! ciliary part
relationship: capable_of_part_of GO:0016055 ! Wnt signaling pathway
relationship: capable_of_part_of GO:0035329 ! hippo signaling
relationship: capable_of_part_of GO:0061512 ! protein localization to cilium
relationship: part_of GO:0035869 ! ciliary transition zone


Add post on maintainer struggles

Great post of experiences over the last 10 years doing oss

🙌 thank you!


im haveing a  bad time running lm-studios

OBSERVATION:
Error condensing thoughts: OpenAIException - Error code: 401 - {'error': {'message': 'Incorrect API key provided: lm-studio. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
Exited before finishing
^CReceived signal 2, exiting...
INFO:     connection closed
ERROR:    Traceback (most recent call last):
File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run
File "uvloop/handles/poll.pyx", line 216, in uvloop.loop.__on_uvpoll_event
File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 66, in uvloop.loop.Handle._run
File "uvloop/loop.pyx", line 397, in uvloop.loop.Loop._read_from_self
File "uvloop/loop.pyx", line 402, in uvloop.loop.Loop._invoke_signals
File "uvloop/loop.pyx", line 377, in uvloop.loop.Loop._ceval_process_signals
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/opendevin/server/session/manager.py", line 45, in handle_signal
exit(0)
File "", line 26, in call
SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/routing.py", line 741, in lifespan
await receive()
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 137, in receive
return await self.receive_queue.get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/queues.py", line 158, in get
await getter
asyncio.exceptions.CancelledError
ERROR:    Exception in ASGI application
Traceback (most recent call last):
File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run
File "uvloop/handles/poll.pyx", line 216, in uvloop.loop.__on_uvpoll_event
File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 66, in uvloop.loop.Handle._run
File "uvloop/loop.pyx", line 397, in uvloop.loop.Loop._read_from_self
File "uvloop/loop.pyx", line 402, in uvloop.loop.Loop._invoke_signals
File "uvloop/loop.pyx", line 377, in uvloop.loop.Loop._ceval_process_signals
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/opendevin/server/session/manager.py", line 45, in handle_signal
exit(0)
File "", line 26, in call
SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 240, in run_asgi
result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/middleware/errors.py", line 151, in call
await self.app(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/middleware/cors.py", line 77, in call
await self.app(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/routing.py", line 373, in handle
await self.app(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/routing.py", line 96, in app
await wrap_app_handling_exceptions(app, session)(scope, receive, send)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/routing.py", line 94, in app
await func(session)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/fastapi/routing.py", line 348, in app
await dependant.call(**values)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/opendevin/server/listen.py", line 40, in websocket_endpoint
await session_manager.loop_recv(sid, agent_manager.dispatch)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/opendevin/server/session/manager.py", line 37, in loop_recv
await self._sessions[sid].loop_recv(dispatch)
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/opendevin/server/session/session.py", line 26, in loop_recv
data = await self.websocket.receive_json()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/websockets.py", line 157, in receive_json
message = await self.receive()
^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/starlette/websockets.py", line 49, in receive
message = await self._receive()
^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 379, in asgi_receive
data = await self.recv()
^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/websockets/legacy/protocol.py", line 552, in recv
await asyncio.wait(
File "/usr/lib/python3.11/asyncio/tasks.py", line 428, in wait
return await _wait(fs, timeout, return_when, loop)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/tasks.py", line 535, in _wait
await waiter
asyncio.exceptions.CancelledError
(devin_env_311) soundmoving465@bryantmorrismin:/mnt/c/Users/zombi/Downloads/GPT/OpenDevin$ kill -9 14831
-bash: kill: (14831) - No such process
(devin_env_311) soundmoving465@bryantmorrismin:/mnt/c/Users/zombi/Downloads/GPT/OpenDevin$ kill -9 25628
-bash: kill: (25628) - No such process
(devin_env_311) soundmoving465@bryantmorrismin:/mnt/c/Users/zombi/Downloads/GPT/OpenDevin$ make setup-config
Setting up config.toml...
Enter your LLM Model name (see https://docs.litellm.ai/docs/providers for full list) [default: gpt-4-0125-preview]: TheBloke/CodeLlama-13B
Enter your LLM API key: lm-studio
Enter your LLM Base URL [mostly used for local LLMs, leave blank if not needed - example: http://localhost:5001/v1/]: http://localhost:1234/v1
/bin/sh: 2: [[: not found
Enter your LLM Embedding Model
Choices are openai, azureopenai, llama2 or leave blank to default to 'BAAI/bge-small-en-v1.5' via huggingface

local
Enter your workspace directory [default: ./workspace]:
(devin_env_311) soundmoving465@bryantmorrismin:/mnt/c/Users/zombi/Downloads/GPT/OpenDevin$ make start-backend
Starting backend...
/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/pydantic/_internal/fields.py:151: UserWarning: Field "model_id" has conflict with protected namespace "model".
/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/pydantic/_internal/fields.py:151: UserWarning: Field "model_id" has conflict with protected namespace "model".
LLM_MODEL="TheBloke/CodeLlama-13B"
LLM_API_KEY="lm-studio"
LLM_EMBEDDING_MODEL="local"
WORKSPACE_DIR="./workspace"

model id has a confict when i run it and look for my model it no where to be found and im running the back end this is my config file
this is the the error that i get and its also saying when i load it up that trying to use it with lm studios

Have you succeeded in the past with this LLM_MODEL setting? I am not sure it works, but lm-studio is openai compatible from what I understand, so it might work with:
LLM_MODEL="openai/CodeLlama-13B"
assuming that the second part of the name is exactly right. The rest unchanged. Can you give it a try?

Have you succeeded in the past with this LLM_MODEL setting? I am not sure it works, but lm-studio is openai compatible from what I understand, so it might work with: LLM_MODEL="openai/CodeLlama-13B" assuming that the second part of the name is exactly right. The rest unchanged. Can you give it a try?

this is my first go around  il give it a shot and see how it works if you have any reconditions to this problem it would be greatly welcome (devin_env_311) soundmoving465@bryantmorrismin:/mnt/c/Users/zombi/Downloads/GPT/OpenDevin$ make start-backend
Starting backend...
/mnt/c/Users/zombi/Downloads/GPT/OpenDevin/devin_env_311/lib/python3.11/site-packages/pydantic/_internal/fields.py:151: UserWarning: Field "model_id" has conflict with protected namespace "model".
You may be able to resolve this warning by setting model_config['protected_namespaces'] = ().

That is a warning, it's annoying but it doesn't affect functionality. It is already tracked and will be fixed.

openai/CodeLlama-13B

After make run, it seems that the front end will forcefully read the model in the settings, and this model cannot be downloaded in ollama, which seems to lead to an endless loop with no solution.

I just put a call out for better LM studio docs. Seems like a lot of folks are having trouble.
Going to close this in favor of https://github.com/OpenDevin/OpenDevin/issues/419


Add build_all script to Builds

This script will compile rippled, run unit tests and then delete all build artifacts in many possible configurations (in Linux).
-j4 is added to make it compile a bit faster. All commands are chained via &&, so if any of them fail, the script will stop at the error.
There are a couple more configurations possible by combining switches by the way, however if even just these would compile + run unit tests without any issue/error/warning for every release, it would already be amazing. :-)
I know that Test.py also exists, that one seems to build much fewer variants though and does not clean up between builds. I am not too much into shell scripting, but I guess this could also be rewritten in a few for-loops, if you are worried that it might slowly deprecate over time (like Test.py).

[Current coverage][cc-pull] is 68.65%

Merging [#1699][cc-pull] into [develop][cc-base-branch] will decrease coverage by -<.01%

2 files (not in diff) in src/ripple/core/impl were modified. more

Misses +9
Hits -9

2 files (not in diff) in ...rc/ripple/beast/nudb were modified. more

Misses +3
Hits -3

3 files (not in diff) in src/ripple were modified. more

Misses -2
Hits +1

@@            develop      #1699   diff @@
==========================================
  Files           862        862
  Lines         61497      61496     -1
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits          42225      42214    -11
- Misses        19272      19282    +10
  Partials          0          0

[][cc-pull]

Powered by Codecov. Last updated by [e9f924c...21f01cd][cc-compare]
[cc-base-branch]: https://codecov.io/gh/ripple/rippled/branch/develop?src=pr
[cc-compare]: https://codecov.io/gh/ripple/rippled/compare/e9f924ca31eb35614ea0ad9fb890f8b20d615dc1...21f01cd60537bbf3ee3bc272cc901b8245479b5d
[cc-pull]: https://codecov.io/gh/ripple/rippled/pull/1699?src=pr

Any comments, anyone actually interested in this?

Thanks @MarkusTeufelberger
I'm sure it's appreciated, just that people have their noses down so to speak.

@nbougalis @ximinez

There's a custom of tagging two reviewers for pull requests via @. I pinged two rippled team members for you :)

Yeah, since your team roles are not public, it's kinda hard to guess for me who is responsible for builds. ;-)

Yeah, just trying to explain/excuse the (as yet!) lack of response :)

There's a lot of overlap here with Builds/Test.py.

@MarkusTeufelberger Thought I'd share a bash snippet. This will get the number of physical cores: num_procs=$(lscpu -p | grep -v '^#' | sort -u -t, -k 2,4 | wc -l) # pysical cores
That lets you do -j${num_procs} rather than hard coding 4.

@ximinez - yes and no. "Test.py --all" just builds "debug" and "release" for every available compiler. Not coverage, not the nounity variants, not profile and it doesn't know about extra options (and their combinations) at all. It (as I wrote above) also does not run the equivalent of "make clean" which actually seems to be an issue with the (failing for me) ninja builds. It does however also run npm tests... though I think you are currently working on replacing them with pure C++ ones, so I didn't add that so far.
@seelabs - I tried something like that but ran out of memory on some builds (I used a simpler version so it detected 8 cores instead of the 4 physical ones due to hyperthreading), thus the artificial static limit. Thanks for the suggestion, I'll add it in since it reports only the 4 physical cores on my machine.

@MarkusTeufelberger I know Test.py doesn't do exactly this. What I'm suggesting is that instead of creating a new script that isn't cross platform, and only works if you have both compilers installed, you modify Test.py to add the features you need.
Examples:

--universe - Build debug release coverage profile
--clean - Remove the build folder in between each target. Somehow.
--ninja - Run ninja after scons.
--nonpm - Skip running the npm tests.

You don't need to implement anything --static, --assert, or --sanitize, since those can already be passed through to scons.

I also thought about extending Test.py and making this script just call Test.py to build all possible combinations or even extending Test.py to a point where "--all" really builds everything. Would you prefer this approach?
The issue I have from time to time is that rippled breaks on some builds but not others and I'm not 100% sure if it is an issue on my end (e.g. the recent switch to gcc 6.1 on my machine messed up Boost pre-1.61 and https://llvm.org/bugs/show_bug.cgi?id=23529 is a constant source of pain...) or if it is something that needs to be fixed upstream. I thought if I gave you an easy way to run all builds + tests, you might be more motivated to run them before tagging releases at least (running this on Travis would be overkill I guess).

@MarkusTeufelberger Yeah, I would prefer something that works with what we already have. ie. Test.py.
If you're getting crashes and/or segfaults when running the unit tests, it would be great if you could send us more information about them. I suspect the problem is more likely to be on our end than yours, since you've successfully been building rippled for so long. Building more configs on Travis is probably a non-starter, since it takes so long already.

Lately it's been mostly warnings or complete build failures on my end (e.g. rippled won't build at all until the new boost version is available, since it is the only one that builds with gcc 6.1). If I ran into segfaults or similar, I'd of course report them.
Alright, I'll work over Test.py a bit, I like Python more than bash anyways... :-)

Alright, I changed it a bit to reduce potential shadowing conflicts and apparently Test.py already does build unity and nounity by default. I added the power sets of possible options (afaik. you can't set both sanitizers at the same time?) to --all and a (preliminary) --clean switch.
You might(?) be able to shoot yourself in the foot if you try hard enough by defining build options yourself when invoking it with --all enabled, other than that it should at least be ready for some sweet compiling action.
Oh well, back to my actual homework. :-/

Thanks @MarkusTeufelberger - I like --clean a lot (I find myself doing rm -rf build/ a lot and I may or may not have accidentally removed Builds instead.

@MarkusTeufelberger This is definitely more on the right track! Unfortunately, the new build-all it isn't working very well on Windows - some of the build types aren't supported, and Windows scons seems to be doing something weird with the extra command line options. I've started fixing the issues as I've found them, but won't be able to put much time into this today. In the meantime, here's my working branch, so you can pick it up if you have access to a Windows build box, otherwise I'll get back to it later today or tomorrow. https://github.com/ximinez/rippled/commits/mt-build-all

Oh, one other suggestion. Why not support --clean in the regular builds, too?

I tried building rippled for Windows a few years ago, found out that the performant database backends (back then) were Linux only and installing VS is close to installing a whole different operating system on top of Windows... so I never really went back there (also because I couldn't find reasonably fast disk pooling options for Windows, so the storage backend is another issue). Is rippled on Windows really a thing out in the wild? The only package (after several years of rippled development) is RPM and I doubt that many people would bother to compile it on their own for Windows.
Build types not being supported (e.g. --sanitize=address on Windows) would probably just make scons itself fail with an error message I guess/hope. I'm not so sure if there is some proper way of getting a list of actually supported build features out of scons besides just trying or hardcoding them in some way (Test.py has no idea about compiler detection for starters). Maybe it would be easier to just use the -k switch when testing all possible builds?
To support --clean in regular builds, I'd need to rework Test.py a bit more, which I'll do probably over the weekend. Currently I tried to keep the existing code path intact and just add my own stuff in a different function to ensure it acts the same, in case some of your internal work flows depend on it. If a more or less complete rewrite/refactoring is ok with you, I'd also get rid of the globals for example.
I'll also add a --nonpm/-n switch to only run the C++ unit tests instead of both npm and C++, since I currently seem to run into issues with file descriptor counts and outdated/deprecated packages in the deeper dependencies of some npm modules.

I run rippled on Windows almost exclusively and it works great with NuDB on SSD.

I'm fine with this. :+1:

@MarkusTeufelberger It's not perfect, but I've made some changes to make it work under Windows (mainly by stripping stuff out), and resolving some issues under linux (Ubuntu). I still haven't had a complete successful run through, partly because it takes FOREVER. Have you?

For me it usually fails on some builds (I'm on Manjaro [basically Arch] Linux) by default because of Clang's/GCC's abi_tag issues, static libraries not being shipped while being hardcoded by scons, Boost going nuts with a new compiler or ninja builds failing alltogether because they don't seem to build the protobuf files for me.
The original (shell) script went through once for me overnight, but I had to disable several builds that were problematic on my platform, as noted above. I then wanted to give you guys the script first and afterwards maybe spin up a Ubuntu Docker instance to see if the issue is my distro (as you officially only support Ubuntu LTS) or the SConstruct file. I have quite a bit to do at the moment, so I unfortunately didn't get to it yet, but I'll still work on this pull request on the upcoming weekend. If you improved the script in the meantime (especially Windows, which I can't test), by all means please feel free to share your changes! :+1:
Running it all will take a while for sure, such is the price for having lots of options. Still I think it is important to actually run everything that you want to support, even if it might be slight overkill or rather used as a "hindsight" tool (e.g. run it on every beta release, but don't delay the release until you've finished the run - but be prepared to push out a new release or deliver quick patches if the build breaks). I would expect that at least RC versions had been built and unit-tested in every possible configuration. Because it is probably cumbersome to do so manually, I think this script is a reasonable way to just fire up in the evening and check the next day.
Another use case is probably checking if/how rippled builds on various Linux distros. Maybe one day you want to provide packages not only for RHEL...? Here a possible Docker option might come into play that fires up vanilla containers, installs dependencies and then tries to build/test everything here.

Oh, shoot, I forgot to include the link to my commit with the Windows changes. https://github.com/ximinez/rippled/commit/e0e4896f0ebdb0bf5debc28472e73ad5774d1a83

About build times: I checked and on my machine with -j4 it takes about 4-5 minutes for a full build from scratch + unit tests + npm tests. With a total of 16 (compiler + target + [no]unity) * 24 (all combinations of options) * 5 (minutes) this means that a full run of all possible builds will take about 32 hours. About half of that time is unit tests, the other half is compiling, so it might even be possible to cut this time in half just by compiling the next binary while still testing the current one. Otoh I'm using a stock i5-2500k, a newer PC with more cores and RAM might be faster anyways. Also this can be distributed anyways if time is critical, build + test are quite closed units.

Only "issue" for now that I'm aware of: The dot in the windows regex is not escaped, but it just matches the actual dot in its place.
Other than that I'd say the Test.py file can now be considered complete, I haven't tested it very extensively, but so far it seems to do what it is supposed to do, also on Python3 (bytes needed to be decoded to become strings).

I think we've finally got it. :+1:

Merged as 55377c12d34bed58daf0c10ce66b334f9eeeea4a.


[RFC] Add a setting to determine skip common chunks behaviour

This is untested code but I am wondering if there is an appetite for a feature like this which would allow you to set skip_common_chunks per config without having to add skip_common_chunks=True to all of your render_bundle template tags.
I realize this is likely an edge case but it would be helpful to have it as a setting instead of solely a template tag option.

Thanks. Please feel free to add this global setting behavior with tests too. Please test if the skip_common_chunks arg properly overrides the settings in all cases.
In near future, we can make this the default behavior, as it is on Webpack's own html-webpack-plugin.

Ok, that sounds great, I will put something together!

@fjsj I've reworked this PR, let me know what you think of this approach.

@fjsj Just wanted to bump this since it's been a couple of weeks.

Thanks, I'll be checking this in the next couple of days.

Released on 1.6.0. Thanks!

Thanks so much @fjsj!


Add timestamps and audit trail to pivots

Discovered when attempting to track changes to BelongsToMany relations. Because we don't have timestamp fields, we cannot know if an attach/detach action has been performed.
Note: For pivots without additional fields, I believe this will still be indeterminable in the api but can still have utility as these models can be queried - and we can even go as far as opening endpoints for them if needed for some reason.
There's also no reason to not maintain an audit trail, though I am uncertain if the nova audit log can be added to pivots.

The auditing package requires a primary key for the auditable type, and pivot types have a composite key, so auditing for pivots for will tabled for now.
Confirmed population of created_at/deleted_at timestamps on new staging seed effort. Closing.


homebrew is no longer accepting formulas at a tap's repository’s root

brew doctor output
Warning: Found Ruby file outside homebrew/jchong tap formula directory.
(/opt/homebrew/Library/Taps/homebrew/homebrew-jchong/Formula):
  /opt/homebrew/Library/Taps/homebrew/homebrew-jchong/test.rb

accordingly to the doc here:
https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap
Tap formulae follow the same format as the core’s ones, and can be added under either the Formula subdirectory, the HomebrewFormula subdirectory or the repository’s root.

Looks like the latest homebrew 4.0.4 no longer looks at repository's root anymore

Verification

[X] My "brew doctor output" above says Your system is ready to brew. and am still able to reproduce my issue.
[X] I ran brew update twice and am still able to reproduce my issue.
[X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

brew config output
HOMEBREW_VERSION: 4.0.4
ORIGIN: https://github.com/Homebrew/brew
HEAD: 4d0154fa7926e8a7b06e9b074dd45957e820407c
Last commit: 3 days ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: a866d1fb067b15d35a3956fef415da3a324d80d8
Core tap last commit: 7 days ago
Core tap branch: master
Core tap JSON: 02 Mar 04:53 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 14.0.0 build 1400
Git: 2.37.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 7.86.0 => /usr/bin/curl
macOS: 13.2.1-arm64
CLT: <IP_ADDRESS>.1.1668646533
Xcode: 14.1
Rosetta 2: false

What were you trying to do (and why)?
I was trying to add my own tap and notice formula at repository's root is no longer accepted but the doc still claim it does.
What happened (include all command output)?
simply cannot find the formula at repo root
What did you expect to happen?
brew should able to find formulas at repo root, or doc should be updated
Step-by-step reproduction instructions (by running brew commands)
brew tap homebrew/jchong https://path_to_repo.git
brew doctor

I can't reproduce this, and your reproduction instructions are broken. You don't provide a tap that reproduces this failure.
Here's a tap that places a formula in the repository root: https://github.com/teamookla/homebrew-speedtest.git
This works completely fine for me.
linuxbrew@codespaces-f53ef7:/workspaces/brew$ brew tap teamookla/speedtest
==> Tapping teamookla/speedtest
Cloning into '/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/teamookla/homebrew-speedtest'...
remote: Enumerating objects: 64, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 64 (delta 7), reused 13 (delta 5), pack-reused 43
Receiving objects: 100% (64/64), 10.52 KiB | 1.75 MiB/s, done.
Resolving deltas: 100% (18/18), done.
Tapped 1 formula (13 files, 75.3KB).
linuxbrew@codespaces-f53ef7:/workspaces/brew$ brew doctor
Your system is ready to brew.

I managed to replicate it:
$ brew config
HOMEBREW_VERSION: 4.0.4-113-gf3cf599
ORIGIN: https://github.com/Homebrew/brew
HEAD: f3cf59997139ffde323ded74f438ffa7ea5a4441
Last commit: 15 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: b8abee171a65550fe897e0e71d9bf64e542ab500
Core tap last commit: 21 hours ago
Core tap branch: cfengine/fix-dir-perms
Core tap JSON: 24 Feb 03:49 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_BOOTSNAP: set
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :0.0
HOMEBREW_DISPLAY_INSTALL_TIMES: set
HOMEBREW_EDITOR: vim
HOMEBREW_EVAL_ALL: set
HOMEBREW_MAKE_JOBS: 6
HOMEBREW_NO_AUTO_UPDATE: set
HOMEBREW_NO_INSTALL_FROM_API: set
HOMEBREW_TEMP: /home/linuxbrew/.linuxbrew/tmp
Homebrew Ruby: 2.6.8 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8_1/bin/ruby
CPU: hexa-core 64-bit skylake
Clang: N/A
Git: 2.39.2 => /home/linuxbrew/.linuxbrew/bin/git
Curl: 7.85.0 => /bin/curl
Kernel: Linux 5.19.0-31-generic x86_64 GNU/Linux
OS: Ubuntu 22.10 (kinetic)
Host glibc: 2.36
/usr/bin/gcc: 12.2.0
/usr/bin/ruby: 3.0.4
glibc: N/A
gcc@11: N/A
gcc: 12.2.0
xorg: N/A

$ brew tap-new test/tube
Initialized empty Git repository in /home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/test/homebrew-tube/.git/
[main (root-commit) 03041a6] Create test/tube tap
 3 files changed, 90 insertions(+)
 create mode 100644 .github/workflows/publish.yml
 create mode 100644 .github/workflows/tests.yml
 create mode 100644 README.md
==> Created test/tube
/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/test/homebrew-tube

When a pull request making changes to a formula (or formulae) becomes green
(all checks passed), then you can publish the built bottles.
To do so, label your PR as `pr-pull` and the workflow will be triggered.

$ cp $(brew --repo homebrew/core)/Formula/go.rb $(brew --repo test/tube)

$ ls -l $(brew --repo test/tube)
total 20
drwxrwxr-x 2 aho aho 4096 Mar  2 13:39 Formula/
drwxrwxr-x 7 aho aho 4096 Mar  2 13:39 .git/
drwxrwxr-x 3 aho aho 4096 Mar  2 13:39 .github/
-rw-rw-r-- 1 aho aho 3210 Mar  2 13:40 go.rb
-rw-rw-r-- 1 aho aho  247 Mar  2 13:39 README.md

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Found Ruby file outside test/tube tap formula directory.
(/home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/test/homebrew-tube/Formula):
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/test/homebrew-tube/go.rb

And other brew commands can't see it either:
$ brew info test/tube/go
Error: No available formula with the name "test/tube/go". Did you mean testkube?
==> Searching for a previously deleted formula (in the last month)...
Error: No previously deleted formula found.

Sorry, the repo I was testing on was internal, but here's how you reproduce it with the repo you used.
From my testing, seem like if you tap with homebrew/ prefix, then brew will not read repo's root, but if it is anything else, it will.
I can work around by not naming it homebrew/xxx
thanks
$ brew tap homebrew/speedtest https://github.com/teamookla/homebrew-speedtest.git
==> Tapping homebrew/speedtest
Cloning into '/opt/homebrew/Library/Taps/homebrew/homebrew-speedtest'...
remote: Enumerating objects: 64, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 64 (delta 7), reused 13 (delta 5), pack-reused 43
Receiving objects: 100% (64/64), 10.52 KiB | 2.63 MiB/s, done.
Resolving deltas: 100% (18/18), done.
Tapped (13 files, 17KB).
nfml16-jchong:Taps jchong$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!
Warning: Found Ruby file outside homebrew/speedtest tap formula directory.
(/opt/homebrew/Library/Taps/homebrew/homebrew-speedtest/Formula):
/opt/homebrew/Library/Taps/homebrew/homebrew-speedtest/speedtest.rb

From my testing, seem like if you tap with homebrew/ prefix, then brew will not read repo's root, but if it is anything else, it will.

Correct.

I can work around by not naming it homebrew/xxx

Yeh, you shouldn't use these for non-official taps.


Page updates for beta

I have massively overhauled the drv pages to be service standard oriented, the idea being we can use this to capture evidence that everyone can see and question. The landing page is therefore now the service standard, the questions asked, and evidence of how we meet it (still needs to be filled in in places).
I have also added a folder structure to help collate supporting pages into some sort of order.
Happy to disccuss as it is quite a radical change

I can't say I've read all of the copy but the coding looks good to me.

Looks OK but I haven't been able to check all of the tags.


KMS CMK key for DynamoDB Global tabel

Hi wanted to know how can we give KMS CMK for DynamoDB global table as KMS is region specific and in global table TF will create table in another region.

Any update on this question ? In our organization, we only support CMK and we must use CMK for global tables.

hi all - this is not supported by the AWS provider for Terraform yet so it is not possible. You will need to file a ticket upstream at the Terraform AWS provider to get it added there https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dynamodb_table

AWS Terraform provider added support for this:
https://github.com/hashicorp/terraform-provider-aws/pull/18373


CFP: Support default deny policy with no rules

As of Cilium v1.16, in order to apply a default deny policy, a CNP or CCNP must have at least one rule in an egress, egressDeny, ingress or ingressDeny stanza.
Furthermore, in Cilium v1.16 there is a new enableDefaultDeny field which seems like it should be able to be used to enforce a default deny posture without listing any rules in one of the rule allow/deny fields. If this field is specified and describes that default deny should be enabled, but there are no rules, then the network policy does not enforce a default deny.
This CFP proposes that the following network policy should enable a default deny posture even though it does not contain an egress, egressDeny, ingress or ingressDeny field.
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: default-deny
spec:
  endpointSelector:
    matchExpressions:
      - key: "io.kubernetes.pod.namespace"
        operator: "NotIn"
        values:
        - "kube-system"
  enableDefaultDeny:
    egress: true
    ingress: true

I believe that currently this is considered to be an invalid policy as it is not fully formed (does not contain an egress, egressDeny, ingress or ingressDeny field). Though I will also note that the cilium operator's network policy validator does not explicitly identify the network policy above as invalid, it just does not have any effect.

My instinct would be to reject empty policies at the apiserver. There should probably be a OneOf directive for Egress / EgressDeny / Ingress / IngressDeny stanzas.
One of the trickier things to understand in the Kubernetes style of network policy is the effect of empty fields. Does it select everything or nothing? Will setting this field change some sort of hidden local variable?
So, I'm generally in favor of preventing "should be a no-op" entry, much how Linux usually rejects unknown flags rather than ignoring them.

That's certainly also an option. Personally when I read the YAML snippet posted in the issue description, the intent seems crystal clear to me.. though admittedly it could bring up some other semantic questions that are less obvious.
For instance, if we allowed there to be no ingress/egress stanzas, would that mean that the following also valid CNP? What does it mean?
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: default-deny
spec:
  endpointSelector:
    matchExpressions:
      - key: "io.kubernetes.pod.namespace"
        operator: "NotIn"
        values:
        - "kube-system"
  enableDefaultDeny: {}

I would interpret this one to be a complete no-op, and therefore I think it'd be helpful to reject this policy. Maybe that's speaking a bit to your instinct @squeed about requiring one of the Egress / EgressDeny / Ingress / IngressDeny stanzas.

If the enableDefaultDeny field in a Cilium Network Policy (CNP) is set to {} (an empty object), two solutions are possible:

Strict validation with status updates
Treat an empty enableDefaultDeny as an invalid configuration. Reject the policy application, update the CNP status to reflect the validation failure, and provide a clear error message indicating the missing ingress and egress settings.

Defaulting to ingress: false, egress: false.
This means that default-deny behavior is disabled for both ingress and egress traffic. The policy would be applied without error, providing a smoother user experience.

While both are viable, I personally favor Option 2, defaulting to disabling default-deny for both ingress and egress.

@renyunkang are you considering the state of the enableDefaultDeny: {} field with or without rule statements like ingress, egress, etc?
For existing policies written without the enableDefaultDeny field, the behavior must not change. This is equivalent to installing the resource without the enableDefaultDeny field, and then if you retrieve the resource with the new schema, it translates the field to the empty value, enableDefaultDeny: {}. Since the existing policies would enable a default deny when there is at least one rule in an ingress, egress, etc stanza, this means that the enableDefaultDeny: {} field must not disable default deny for each direction. In order to disable default deny, you must explicitly set enableDefaultDeny: ingress: false (and/or equivalent egress statement).
As a general constraint of the system, if some configuration has no effect, then I think it can improve user experience for the resource to be rejected. For example the resource described here. This helps to communicate when the user has created a configuration that does not do what they intend to do.

To align with @squeed's statements above, I think it would be simpler to just require a rule to be specified. Effectively that means there would be a small change from your post above @renyunkang :
If the daemon has EnableNonDefaultDenyPolicies set to true:

One of the following must be configured: Ingress, IngressDeny, Egress, EgressDeny, ~EnableDefaultDeny.Ingress, or EnableDefaultDeny.Egress~.

One of the benefits is that for this simpler expression it should be easier to embed this in the CRD validation so that the kube-apiserver enforces it, rather than having the cilium-agent enforce it after the resource is already configured. Better UX.

To align with @squeed's statements above, I think it would be simpler to just require a rule to be specified. Effectively that means there would be a small change from your post above @renyunkang :
If the daemon has EnableNonDefaultDenyPolicies set to true:

One of the following must be configured: Ingress, IngressDeny, Egress, EgressDeny, ~EnableDefaultDeny.Ingress, or EnableDefaultDeny.Egress~.

One of the benefits is that for this simpler expression it should be easier to embed this in the CRD validation so that the kube-apiserver enforces it, rather than having the cilium-agent enforce it after the resource is already configured. Better UX.

The current description completely contradicts the CFP title. Is this modification consistent with the CFP proposal? Or should we revise the CFP proposal title?

With the following rule configured, what happens to ingress traffic?
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
spec:
  endpointSelector:
    matchExpressions:
      - key: "io.kubernetes.pod.namespace"
        operator: "NotIn"
        values:
        - "kube-system"
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  enableDefaultDeny:
    egress: true
    ingress: true

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
spec:
  endpointSelector:
    matchExpressions:
      - key: "io.kubernetes.pod.namespace"
        operator: "NotIn"
        values:
        - "kube-system"
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  enableDefaultDeny:
    egress: true
    ingress: false

Will setting enableDefaultDeny.ingress to true or false allow all ingress traffic in either case? @joestringer  @squeed

The current description completely contradicts the CFP title. Is this modification consistent with the CFP proposal? Or should we revise the CFP proposal title?

Sorry, I agree it contradicts the CFP title. I created the proposal to try to gather feedback but I don't hold a strong opinion that my original description of the solution is the right way. Probably if I were to rewrite the issue, I would reframe it to first describe:

The problem: it's confusing that enableDefaultDeny.egress/enableDefaultDeny.ingress by itself has no effect
A solution: my initial proposal was to change the semantics so that it takes effect.
A different solution: @squeed's counter-proposal was that we just declare such policies to be invalid.

Your questions are super helpful to help us reason about what we think the right solution to the problem is. I think that the logical conclusion of Casey's proposal would be that the network policies you proposed in your latest post should both be rejected, because they configure the ingress default deny posture but they do not contain any ingress rules.

Okay, based on your description, let's elaborate with an example: @joestringer  @squeed
The problem:
Before v1.16.0, the following rule allowed all ingress traffic:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
spec:
  endpointSelector:
    matchExpressions:
    - key: "io.kubernetes.pod.namespace"
      operator: "NotIn"
      values:
      - "kube-system"
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"

After v1.16.0 introduced enableDefaultDeny, the above configuration, regardless of whether enableDefaultDeny.ingress is true or false, still allows all ingress traffic. This is confusing for users because the configuration has no effect.
Proposed Solutions:
Solution 1: Change the semantics of enableDefaultDeny so that it takes effect as intended.
Solution 2: Declare the policy invalid.
My Understanding:
Solution 1 is easy to understand.
How should Solution 2 be interpreted? Detecting invalid policies works well for policies like this:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
spec:
  endpointSelector:
    matchExpressions:
    - key: "io.kubernetes.pod.namespace"
      operator: "NotIn"
      values:
      - "kube-system"
  enableDefaultDeny:
    egress: true
    ingress: true

However, Solution 2 might not be suitable for the first example. Should we still use Solution 2 for the first example? Since there's no ingress or ingressDeny section, enableDefaultDeny.ingress cannot be configured, right?

Your questions are super helpful to help us reason about what we think the right solution to the problem is. I think that the logical conclusion of Casey's proposal would be that the network policies you proposed in https://github.com/cilium/cilium/issues/35558#issuecomment-2503006466 should both be rejected, because they configure the ingress default deny posture but they do not contain any ingress rules.
For existing policies written without the enableDefaultDeny field, the behavior must not change.

Furthermore, if setting enableDefaultDeny.ingress to either true or false results in the rejection of all ingress traffic, will the behavior change for policies without the enableDefaultDeny field after users upgrade from v1.15.x or lower versions?

It's a bit nuanced to avoid severely breaking existing clusters. We can think of the default deny configuration as a tri-state for each direction: It is either not present in the policy (v1.15-compatible), or it is present with a true (deny all by default) or false (do not deny all by default; only apply the rules explicitly listed in the policy) value.
This is how I think that Solution 2 solves these problems:
If there is an ingress or ingressDeny stanza with at least one rule:

enableDefaultDeny.ingress: true or no enableDefaultDeny section must place the subject endpoints into default deny, ie reject all ingress traffic that does not match the rules.
enableDefaultDeny.ingress: false must not place the subject endpoints into default deny. Only the traffic clearly selected by the rules will be impacted by the rule.

If there is no ingress or ingressDeny stanza:

If there is enableDefaultDeny.ingress with any value: Reject the policy. No change in enforcement, this should ideally occur when the resource is created or at worst case cilium-agent rejects the policy.
If there is no enableDefaultDeny.ingress or enableDefaultDeny stanza: no change.

If there is no enableDefaultDeny field:

No change vs. v1.15 policy enforcement or network policy parsing.

Then the equivalent should apply for egress / egressDeny.


Adding support for float infinities in C++ backend

Currently the C++ backend will throw a NotImplementedException whenever a Value evaluates true for the following:
Double.IsInfinity(Value) || Double.IsPositiveInfinity(Value) || Double.IsNegativeInfinity(Value)
I'm not sure why the code was testing for the 3 possibilities here. Anyway, should we have some method in the runtime to provide those values to the C++ backend? If so, what is recommended way for C++ code? I know the hex representation of those values, but it might be better to rely on some other mean.

I think that the easiest way to deal with this is to take pointer to the bit representation that came from the IL instruction and recast it as double.
The BCL depends on the bit representation of floats and doubles to be the regular IEEE in number of places, so taking a dependency on it in the C++ code should be ok.

To make this easier, it would be good to have a helper function like this one:
double __int64_to_double (int64_t val)
{
    union
    {
         int64_t i64;
         double d;
    } t;
    t.i64 = val;
    return t.d;
}

somewhere in the code. Are you ok with this and if so where should we put it?

I would start a new file, something like - src/Native/Bootstrap/CppCodeGen.h/cpp - and start adding the random small CppCodeGen helpers to it. E.g. INT64VAL or CORERT_UNREACHABLE  that you have added recently can be move to it as well.

Sounds good. For now, I can get away with an inline and just a .h file.

Submitted PR #1164

Closed via PR #1164


YARN-11465. Improved YarnClient Log Format

JIRA: YARN-11465. Improved YarnClient Log Format

LGTM.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 53s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
39m 29s

trunk passed

+1 :green_heart:
compile
0m 37s

trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1

+1 :green_heart:
compile
0m 34s

trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09

+1 :green_heart:
checkstyle
0m 34s

trunk passed

+1 :green_heart:
mvnsite
0m 39s

trunk passed

+1 :green_heart:
javadoc
0m 40s

trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1

+1 :green_heart:
javadoc
0m 28s

trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09

+1 :green_heart:
spotbugs
1m 10s

trunk passed

+1 :green_heart:
shadedclient
20m 21s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 27s

the patch passed

+1 :green_heart:
compile
0m 28s

the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1

+1 :green_heart:
javac
0m 28s

the patch passed

+1 :green_heart:
compile
0m 26s

the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09

+1 :green_heart:
javac
0m 27s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 19s
/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 91 new + 56 unchanged - 0 fixed = 147 total (was 56)

+1 :green_heart:
mvnsite
0m 27s

the patch passed

+1 :green_heart:
javadoc
0m 22s

the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1

+1 :green_heart:
javadoc
0m 21s

the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09

+1 :green_heart:
spotbugs
0m 56s

the patch passed

+1 :green_heart:
shadedclient
20m  5s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
28m 16s

hadoop-yarn-client in the patch passed.

+1 :green_heart:
asflicense
0m 40s

The patch does not generate ASF License warnings.

119m 55s

Subsystem
Report/Notes

Docker
ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5550/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5550

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 941b17f8f968 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 8363709d805df0b782dcbf294b3e312a01007f01

Default Java
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5550/1/testReport/

Max. process+thread count
677 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5550/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@yl09099 Thanks for the contribution! merged trunk.


MR Permissions - Group and Project dropdown issues

RHOAIENG-12025
Description

Verify that once you've selected an item in the dropdown, if you then open the dropdown, you can NOW focus the typeahead text field without the need to close the dropdown( it was blocking the user from doing it) - try typing something in the typeahead in addition to the selected option
The Option in the bottom of the dropdown list is now Select and not Create
After you choose an option in the drop down, try deleting it in the typeahead and verify that it's not creating duplicate entries of that field in the dropdown(see screenshot in the issue)

How Has This Been Tested?

Tested on the UI
Test Impact

One test change to update the Create to Select . Also When changing the label to select I didn't change all the variable names of that label so they are still mostly shown as "create ...." in the code and not as select (except at the original line)
Request review criteria:

Self checklist (all need to be checked):

[x] The developer has manually tested the changes and verified that the changes work
[x] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[x

] The developer has added tests or explained why testing cannot be added (unit or cypress tests for related changes)

If you have UI changes:

[ ] Included any necessary screenshots or gifs if it was a UI change.
[ ] Included tags to the UX team if it was a UI/UX change.

After the PR is posted & before it merges:

[ ] The developer has tested their solution on a cluster by using the image produced by the PR to main

So I ended up removing the stuff you suggested to remove and not adding the code you proposed because it looks like it was creating this pre selected empty field. At the current state everything looks like it's working as desired, let me know what you think

@YuliaKrimerman without the code I proposed, when I type "istio" and click the "Select 'istio'" option, the dropdown appears as if nothing is selected even though the value has updated (the confirm button in the row becomes enabled).
I do notice the issue you pointed out with my suggestion, it's because I forgot to make sure value is defined before pushing it as an option. This should fix it:
  // If we've selected an option that doesn't exist via isCreatable, include it in the options so it remains selected
  if (value && !selectOptions.some((option) => option.value === value)) {
    selectOptions.push({ value, content: value });
  }

@mturley Yep, you're right, now it works good. Pushed the latests and ready for re-review


Is it possible to change the active color for icons

Thanks for implementing this, it's been great so far. I'm wondering how to set the active/inactive colors for icons. For TabBarIOS, there was a tintColor prop. So for the below example, how would I change the icon for the A tab to white instead of blue? Thanks in advance for your help.

The TabItems have a prop called renderSelectedIcon -- you could return a white image (use tintColor to help you) from there.

Perfect - thanks for your help.


Control of chart animation

It would be great to have the ability to control the animation of the data being written to the charts. For example a line graph that would draw temperature in realtime while the use is watching it.

Hello, @thekeenant, first of all, thank you for your work!
Probably not what the author meant, but let me elaborate on this issue: is there a way to control animation duration?
Let's say I use LineChart, and I feed new data every second. I want the animation to go slow, and smooth so it doesn't seem like it stops between states, for that I would need to control animation time.

@thekeenant Just to clarify, I mean animation which happens on changing DoubleSpan.


⚠ Jenkins Plugins Artifactory has degraded performance

In 5d3ee3f, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 2804 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 76346df after 24 minutes.