NetworkedMultiplayerSteamP2P

I have read this issues #145 #124 and saw this repo https://github.com/JDare/GodotSteamHL but I still don't quite understand what exactly is the problem with integrating HLAPI with Steam's networking sockets directly with a dedicated peer on a module-level. Most of what I came across the internet (including mentioned repo) is some basic lobby system setups to pass arbitrary streams of data in-between by using self-made protocols e.t.c. which is not that hard to set up in the first place. But what about the usage of Godot's multiplayer API (slave/puppet/RPC stuff/position sync/attribute sync/etc)? All that essentially needs to be done is the replacement of data sends-receive/ connect functionality from ENet to the SteamAPI, while still having a fully backed by the engine serialization/synchronization functionality provided by high-level API. @Gramps I would really appreciate some feedback on this since all of the classes mentioned in those issues as far as I know are already in place. What would be the most reasonable way to implement something like this on the current version of the module, since most of the required functionality is now accessible to the users of GodotSteam?

My understanding of networking is pretty basic.  And I will also note that the newer Steam networking classes will be replacing the current networking class and P2P stuff.  The newer stuff is still in beta, last I checked, and not fully implemented in GodotSteam yet.
That being said, I have still yet to investigate mixing Godot's HL networking with Steam's fully; way too much on my plate still. My off days usually get overwhelmed by one random thing or another.  I think that mixing the two might just be overkill though; more code than needed to do the job.  To me it makes more sense to use one or the other; and if your game is on Steam might as well take advantage of their networking features instead of coding it all yourself in Godot's.
I think one of the original problems was that Godot's relied on IP address where as Steamworks does not; Steam handles all the address and port stuff if you use their functions correctly.  I had an example of passing the IP through Steam's invite system but it was kinda janky.  My last Godot 2 game did this and it was messy but functional.  Also, I think the newer networking stuff does more to expose the IP and port data though.  In my personal projects I have started working more with networking and authentication, but strictly using Steamworks functions and no Godot networking.  It seems a little more straight-forward than the Godot HL networking; but I may be biased or somewhat uninformed.
Other than in tutorials, nothing from Godot's networking will be mixed into the module; it should stay as close to Steam's native implementation as possible.  If I or someone else can properly mix the two and write up a tutorial, it'll get added.  Like I said, I plan to tackle it at some point.

Nonetheless, I have it on my list to review and it'll happen at some point.  Just might be a little while.

Thanks for the response! I really really really need this feature for my game, since using custom protocol is not an option for me (I tried going that way, but the overhead that I had due to the complexity of my game made me write more and more of the networking code then of the game itself), I need to make P2P and HLAPI friends somehow (In this context I mean P2P connections, not P2P as an authoritative model). If it matters, I can finance you for your development time spent on this. (I tried hiring someone at Godot's discord server, but no one has responded so far).

Is there any reason you need to use the Godot HL networking stuff?    If making it strictly P2P with no master/slave, that should be easy to do with Steamworks.

I really don't want to reinvent the wheel with position syncing, scene instantiation sync, the scene itself sync, RPC, object ownership and there is just so much other things that needs to be synced if I am going to use custom protocol, where in HLAPI almost everything that I have mentioned comes pretty much out of the box.

I ended up implementing this using CustomMultiplayer in Godot 3.5. It seems to be working for me as a drop-in replacement for NetworkedMultiplayerEnet, with the exception that you have to manually call poll (which I believe will no longer be a limitation with MultiplayerAPIExtension in godot 4):
https://gist.github.com/rcorre/776926518a962c3cb193e07de1bf61b3
@Gramps would you be intrested in me contributing this to the docs as a gdscript example, or even to GodotSteam itself as a c++ implementation?
You can use it like this:
func _ready() -> void:
	# Could also let tree automatically poll, but I wanted to control the order of polls
	get_tree().multiplayer_poll = false

func _physics_process(_delta: float) -> void:
	if multiplayer.network_peer is NetworkedMultiplayerSteam:
		(multiplayer.network_peer as NetworkedMultiplayerSteam).poll()
	multiplayer.poll()

func start_game():
	var lobby_id := create_or_find_lobby() # However you want to do this
	var peer := NetworkedMultiplayerSteam.new()
	peer.join_lobby(lobby_id)
	get_tree().multiplayer.set_network_peer(peer)
        # now use Godot's RPC system

Hey there!  That is really neat and, I think, in line with what @greenfox1505 has been up to: https://github.com/greenfox1505/GodotSteam/tree/multiplayer-for-godot4
I think merging Steamworks into Godot's networking stuff is a great idea and would love to integrate it into GodotSteam.  I've been super busy but my time is opening back up so I can start looking into this stuff more.  Seems you and @greenfox1505 have laid most of the groundwork already too!
I'm not sure how much the two differ but maybe we all can start a discussion about it.  What say you @rcorre and @greenfox1505?

Multiplayer peers on both Godot3 and Godot4 both will call poll once a peer moves to a connected state. But GodotSteam needs a call back every frame in order to register that you have moved to a connected state.
This is cool though. I'd like to see your code. At some point my project will get moved into master and if there are some useful things from your version, I'd like to include them.

Multiplayer peers on both Godot3 and Godot4 both will call poll every frame when a peer moves to a connected state

Unfortunately not if you're using gdscript since poll is not virtual on the gdscript side. That's not a problem in c++ though.
It would be great to work together to get this implemented though.
@Gramps would you want an implementation for Godot3, or just Godot4? I steered away from the latter just because I wasn't sure if the APIs were stable.

I would say both as 3.x is still heavily used but 4.x eventually will be the main use.  I'm not sure how hard it is to port to both of them.  Since my current game project is coming to a close soon (though I thought it already would have), I need to get more up to speed on all of this soon.
I think 4.x is finally stable enough.  I believe someone was also porting @greenfox1505's stuff into Godot 4x.... right?

Well, I was going to ask if I should bring in my 3.x implementation, and just like that, 4.0 is officially released :laughing:

I'm doing my stuff in godot4.0 now. There is an old version that mostly worked for godot3.x, but everything I'm working on now is godot4.
What I'm working on can totally be backported to godot3.x, if you need it. I can help a bit with that, but it won't be a priority for me.

Having this in 3.x would be awesome but 4.x is the future.  Surely we can do both.  :D

What I'm working on can totally be backported to godot3.x, if you need it. I can help a bit with that, but it won't be a priority for me.

Oh, no worries. I already had a working solution for 3.x, and was asking if I should port it to c++ for GodotSteam. But with 4.0 officially being released, I'm just moving to 4.0.

I managed to get something working for 4.0, here it is in case it's useful: https://github.com/Gramps/GodotSteam/compare/master...rcorre:peer4.0

I will look at this at the end of the week. I have some day job deadlines before GDC and then after that I'm gearing up for my own trip. So I may not have time to review this until at least after that.
https://github.com/greenfox1505/GodotSteam/tree/multiplayer-for-godot4
This is my current project space.  If you've got some fixes and behaviors I don't, we can absolutely merge that in.

Thanks @greenfox1505 , no rush at all! I just wanted to get something to the point where I could continue with my game.
It looks like our implementations ended up pretty similar.

Mine's minimal and doesn't replicate much functionality already in Steam, so I didn't add any signals or functions for e.g. creating lobbies. I figured you'd still use Steam to do that.
Yours has a builtin ping mechanism mine doesn't -- I figured you'd do that via rpc if you wanted one.
You use generate_unique_id while I use the bottom 31 bits of the AccountID. I suppose both have the same probability of a collision, though generate_unique_id is probably better, because if two players do have a collision, they'd be able to retry and it probably would work. The docs don't actually say the peer_id has to be a uint31 any more (they did in 3.5), but based on the code that still looks true. I asked here: https://github.com/godotengine/godot/issues/74628#issuecomment-1463717863
Neither of us implemented reliable ordered rpc (seems doable, but annoying).

I liked generating the id because it's unique to the connection. If you loose connection and have to reconnect, you get a new id. It does have the possibility of collision, but so does enet
The ping functionality I have is purely for keep-alive. A client doesn't register a peer has joined until it gets one ping from it. Then it periodically pings to make sure the connection is still valid. The next step there would be to reestablish the connection on failure.

I was just dropping in to suggest something like this, and so many people have beaten me to it!
My big interest is to use Steam, but not be limited to it—eventually, I might want to drop this on Epic, or another game platform, and I need to maintain that flexibility. I also like the idea of people being able to host their own games locally or without relying on a third party network.
I'm also not stoked about dropping down to GDExtension for the sake of it, because I would need to support multiple platforms (including Mac, which I don't have for testing) and also feel that it should be reasonably standardized... So it's kind of a last-resort thing for me. (I have almost no experience with GDExtension.)
That said, I have a plan here. ENet is how Godot generally does local multiplayer. This relies on ENet being set as the peer, so that MultiplayerSynchronizer and MultiplayerSpawner work out-of-the-box. However, if we set the peer to a dummy (Maybe OfflineMultiplayerPeer), and add a new child node that uses exposed GodotSteam functions to synchronize the chosen properties over Steamworks instead of ENet, we will have a drop-in replacement that depends entirely on configuration.
That's what I'm working on at this point, or at least it's my action item list. We would have MultiplayerSynchronizer->SteamSynchronizer, and MultiplayerSpawner->SteamSpawner; both of which would examine properties of their immediate parent and transmit them over Steamworks, while the parents would remain relatively inert.
This is less flashy than implementing MultiplayerPeerExtension, but it's flexible, insulates me from the hardware level, and could easily be swapped out for, say, some future EpicSynchronzier and EpicSpawner or something like that. It would entirely be a question of the game mode.
Assuming this all holds up, maybe it might solve some outstanding issues for GodotSteam too?

This relies on ENet being set as the peer, so that MultiplayerSynchronizer and MultiplayerSpawner work out-of-the-box

Will MultiplayerSynchronizer and MultiplayerSpawner not work with an arbitrary peer? I haven't tried them yet, but I figured they'd use the rpc system, so as long as you have a MultiplayerPeer that implements RPC they should "just work".

The MultiplayerSpawner and MultiplayerSyncer already work. SteamMultiplayerPeer is only a transport layer. Under the hood, they use the same transport paths as RPCs.

@greenfox1505 I'm sorry, I seem to have totally missed that you're extending it to include a Steam peer. It was a lot to read. Thank you!
I'll try implementing it tomorrow. It's been a little while since GDC, have you made much progress? If it's ready to go, I may dive into it just to get a better feel of how peers work under the hood. (Mostly a multimedia guy, my networking knowledge is tangential.)
Does it need any documentation written, or is there anything I could help with coming in late?

https://github.com/greenfox1505/GodotSteam/tree/multiplayer-for-godot4
My repo is here. My plan was to merge this in when I had finished a game jam project using it, proving its soundness. That jam did not go as planned. Now the plan is to create a simple project that will become a proof of concept for most of what I'm trying to accomplish here.

Sorry your jam didn't work out! That said, I've got an arena shooter going on ENet which I want to publish on Steam, and the ability to simply change the network peer and have it work as usual over Steamworks would be a blast.
I'm going to go ahead and download it, copy my project, and see if I can get this to work with your GodotSteam branch on game ID 480. Is there anything I need to worry about with, say, building for other export templates? I'm running Mint 20.3 Una right now, and I'll also want to be building a Windows executable.
With everything I've done with ENet, I might be able to help with that proof-of-concept project too.

Is there anything I need to worry about with, say, building for other export templates? I'm running Mint 20.3 Una right now, and I'll also want to be building a Windows executable.

@MichaelMacha I'm also developing on Linux. The normal cross compiling instructions for Godot do not seem to work with GodotSteam. Compilation succeeded, but the resulting binary wouldn't start when friends tried to run it on Windows .
I ended up modifying the Github Actions on my fork to produce builds for me (the GodotSteam actions pull the SDK from a private repo).
I do have a release here I've been using with my peer implementation (I had already started on one before I realized there was work in progress).

@rcorre Odd, I'll bear that in mind. I had a few issues getting it to compile a moment ago, until I realized that it was also looking for (I think) the tools directory from the Steamworks SDK. When I unpacked everything into the module, it started to work. All the instructions asked for was public and redistributable_bin.
The example application was probably unnecessary, but hey, I get paranoid when I'm arguing with a compiler.
Time for a break, but I'll keep your repo in mind if I hit flak on a build.

@greenfox1505 Scratch that, I apparently was using the Gramps GodotSteam. On Linux, using your branch, I'm getting a long sequence of incomplete type (generally GetTypeInfo) errors, and ambiguous conversion errors, which prevents the compilation from completing.
It almost sounds like I'm missing a header somewhere, somehow. Most of them are in header files. They all happen when Scons hits steam_multiplayer_peer.cpp. Maybe someone can give me a hint what I'm missing?
I'll drop the compilation report on pastebin for you, if it helps. Everything else has already been built.
Thanks for all your work here.

We're using @greenfox1505's peer (but rebased on a later GodotSteam build) in production with our Early Access game. High level nodes work most of the time - we noticed an issue trying to use a Synchronizer right at the start (we just use RPCs in that case). But that could also be something else going on in our game.
But we've had hundreds of people playing and streaming multiplayer over Steam without issue.
We also offer ENet (just call it Direct IP to players) and Offline - switching between peers is effortless.
This should definitely be the future of GodotSteam's multiplayer approach. Happy to help where I can.

That's super-cool it is working in a big use-case.  What is your game called?

That's super-cool it is working in a big use-case. What is your game called?

Oh you already know it ha - Drift

Ah yes!  Last I played I was drifting off into the void!

I'm also working on a GDExtension SteamMultiplayerPeer class, admittedly mostly just to improve my knowledge of GDExtension, peer networking, and brush up on C++.  You may end up having to cherry pick from the lot of us, Gramps! It's great to see so much interest.
My goal is to get to the point where arena-style multiplayer is just as easy with GodotSteam as it is with ENet or another default option. Just plug in the peer and let the synchronizer worry about it. I seem to be tripping on a missing pure virtual method, though... could be a bit.

Yeah, people have been asking about how to do this for quite some time.  That's awesome you all are working on it.

I'm also working on a GDExtension SteamMultiplayerPeer class, admittedly mostly just to improve my knowledge of GDExtension, peer networking, and brush up on C++. You may end up having to cherry pick from the lot of us, Gramps! It's great to see so much interest.
My goal is to get to the point where arena-style multiplayer is just as easy with GodotSteam as it is with ENet or another default option. Just plug in the peer and let the synchronizer worry about it. I seem to be tripping on a missing pure virtual method, though... could be a bit.

Wow I'm glad you're venturing too, currently I'm just making small tweaks, like using enums that already exist on godotsteam.
Although I think the biggest pain is figuring out if there's any way for steamMultiplayerPeer not to duplicate calls or signals, like create_lobby

@scriptsengineer There's usually an architectural way to work around potential duplicate code. We'll figure something convenient out. I'm thinking maybe something like a SteamMultiplayerPeer constructor that takes the connected lobby ID as a parameter...
The biggest issue I'm looking at is comprehensively understanding GDExtension. I've got a bunch of little things working fine, but extensions of MultiplayerPeer itself insist on being abstract in GDScript. It doesn't look like an abstract class on the C++ end. Am I just overlooking something simple?

@MichaelMacha I may be misunderstanding your question, but in GDExtension I believe you need to extend MultiplayerPeerExtension instead of MultiplayerPeer.
Check godot-cpp/gen/include/godot_cpp/classes/multiplayer_peer_extension.hpp after building in godot-cpp. You'll see that all the methods are virtual (unlike multiplayer_peer.hpp).
(technically it's not

Wow was answering now that, that's exactly what the @rcorre says,
Another update on the implementation of gdextension is that I intend to redo the work of greenfox using steam sockets and not lobbies, at first lobbies is something of high level that the user must implement something related.
See more in discord topic https://discord.com/channels/564589037411893260/1021131693979090994/1115437306086690916

I believe you need to extend MultiplayerPeerExtension instead of MultiplayerPeer.

Oh Lord... I figured it was something ridiculous. @scriptsengineer Thank you sir; GameDev Stack, Stack Overflow, and the Godot forums had nothing for me on this, and you saw the problem immediately. :-)
On the bright side, chasing this awkward bug has given me a profound understanding of how arbitrary parts of GDExtension work under the hood; so I guess there's that.
My explicit goal is to get the data streams already synchronized over ENet (as an example), without any change of their formatting, to flow over the Steam network instead; I'm trying to finick with the minimum possible amount. My goal is to be able to set up a property to be synchronized, and simply have it pipe through Steam instead of another network. As I understand it, that's most of what Peer orchestrates.

I believe you need to extend MultiplayerPeerExtension instead of MultiplayerPeer.

Oh Lord... I figured it was something ridiculous. @scriptsengineer Thank you sir; GameDev Stack, Stack Overflow, and the Godot forums had nothing for me on this, and you saw the problem immediately. :-)
On the bright side, chasing this awkward bug has given me a profound understanding of how arbitrary parts of GDExtension work under the hood; so I guess there's that.
My explicit goal is to get the data streams already synchronized over ENet (as an example), without any change of their formatting, to flow over the Steam network instead; I'm trying to finick with the minimum possible amount. My goal is to be able to set up a property to be synchronized, and simply have it pipe through Steam instead of another network. As I understand it, that's most of what Peer orchestrates.

I'd be happy to see this work in progress, my plan is to do just that by converting enet code to steam sockets, just like that.

@scriptsengineer It's later than heck, but I've successfully gotten an instantiable NullaryMultiplayerPeer going. Thank you. Code is progressing along.

Y'all are doing great work. I'm currently playing around with @cridenour 's implementation and just now managed to get the first basic things to work, though I seem to have issues with the MultiplayerSpawner not syncing?
Maybe @cridenour could elaborate a little on how their implementation is used? Might help other peeps too.
I've been working on a co-op space-horror game the past 2 years, using ENET for the multiplayer stuffs which is working very well now. Then I realized that for the Steam version I needed some sort of relaying through Steam to not have to deal with NAT/ports etc so needless to say what you all are doing has been a lifesaver. Now I don't need to rewrite all of my netcode haha.
And I can also release the game on other platforms if needed! woah!
Looking forward to an official implementation in GodotSteam in the hopefully near future :)

though I seem to have issues with the MultiplayerSpawner not being found?

Does lobby/PlayerSpawner already exist when the connection is initiated or are you changing scenes as part of the connection process? I've found it's more reliable have the scene fully loaded before connecting to other players.

redo the work of greenfox using steam sockets

I had a similar conversation with Faless (who wrote the new Godot nodes) in RocketChat and he agreed that Godot is happier when connections are lower level so it may be a better fit than the Messages API.

Glad to hear it. I never needed to do that with ENet so that's where I hope switching to the sockets and having a clearer connection state to report to Godot will not make that necessary.

Yeah that may be a better idea. Also I noticed that messages seem to arrive less well through Steam. Got quite a few bugs that I don't have with ENet.
Could this possibly be an issue with rpc_id() ?

Whew.
So, I've been working on a SteamMultiplayerPeer extension for (at least) 4.0.3; though I know there are some minor changes to class registration in the coming Godot 4.1. It's primarily based off of @scriptsengineer's work; I've been hand-coding a lot of it to ensure I actually know what's going on. Little bit of cleanup. I spent about six hours on it yesterday and it's almost compiling now.
That said, there are a few odd changes between Godot 3 and Godot 4 which I can't quite make sense of or find adequate documentation on (doesn't mean it isn't out there), and I'm wondering if anybody here can give me a hint.
 I mean usually I love structs and classes that have specific-for-script components, but the docs just tell me to worry about it if _put_packet isn't implemented, which actually sounds like a pretty fundamental function. Is there a preferred approach for that, and does anyone know of an implementation I could review, or could point me to a difference service (other than Steamworks) that does have an open implementation, for my own edification?
Other than that I'm just implementing the getter/setters for transfer mode, transfer channel, new connection acceptance, the constructor, the destructor, and of course _bind_methods (which may be a handful). I'm also doing some dusting for unnecessary semicolons and other final-polish issues, but it may be a bit early for that.
Once it's building, without warnings, I thought maybe I would turn it into a pull request and maybe we could talk about merging it with GodotSteam core or making adjustments?
Again, this is just a MultiplayerPeer class, but if a Steam peer could be brought to the point of being as easy to drop in as ENet or WebRTC it would be wondrous.

 I mean usually I love structs and classes that have specific-for-script components, but the docs just tell me to worry about it if _put_packet isn't implemented, which actually sounds like a pretty fundamental function. Is there a preferred approach for that, and does anyone know of an implementation I could review, or could point me to a difference service (other than Steamworks) that does have an open implementation, for my own edification?

Usually when I want some reference of network peer implementation I see WebRTC do Falless https://github.com/godotengine/webrtc-native
This part also I couldn't really understand the implementation, this is inherited from @greenfox1505 code.

See this for context: https://github.com/godotengine/godot/pull/63049
tl;dr: used for GDScript based peers

Ah, OK, so I probably don't need to worry about overriding it at all. Great.
The other thing bugging me a little, is there were a number of cases where a Steam ID (which is an unsigned long long int) was being packed into a Variant, which doesn't seem to support more than unsigned long int. (I'm writing code on Qt Creator and my internal compiler wasn't happy with it.)
"unsigned long long int" isn't actually a standard-length type, though it's usually 64 bits. Is it possible to jam a 64-bit number into a Variant? There's a good chance that that cast dropped 32 significant bits, and I would rather not depend on that.

I believe that's why @greenfox1505 created a SteamID class in his implementation since you couldn't/shouldn't be passing it through variant at this point.

@cridenour Great news to hear that it's been settled, I'll reference @greenfox1505's workaround before I pull request a compilation. It may compile, but casting to an unsigned long int just feels dangerous.

SteamID has been implemented and compiles; though I'm noticing that it hasn't yet been massaged into the SteamMultiplayerPeer code. That's for either tomorrow or the next time I'm up for a little coding, given that tomorrow's a Sunday.
Unless it totally has been worked in and I'm just mind-blind and missing it in the Github listing right now... in either case, here goes!

@scriptsengineer Yeah, there is an inherent minimum of two standards here; but we should at least be consistent between them if we're building an adapter.

No one's had any issues with it actually finding the Steam API, right? I figured I would take a knock at resolving the STEAM_CALLBACK bugs, which boil down to a "no matching function for call" bug. There are a few reasons that could be. It looks like the SConstruct file does a decent job finding the actual library, but I thought I should ask just in case.

OK, after a few attempts at compiling with different versions of the library (1.57 and the earlier 1.55, including after cleaning), I can't seem to shake (or understand) these particular errors. I've verified that it's the STEAM_CALLBACK calls (commenting them out turns "no matching function" to "declaration not found" on the implementation, as expected). But, I can't figure out why it isn't finding the functions.
Also I have to admit that I'm a little hazy on what these lines do, exactly. I mean, they certainly look important.
Here's the first error (the others are almost repeats):
/media/michael/External/GodotSteam Test/GodotSteam/godotsteam/steam_multiplayer_peer/steam_multiplayer_peer.cpp:551: error: no matching function for call to 'CCallback<godot::SteamMultiplayerPeer, LobbyChatMsg_t>::CCallback()'
godotsteam/steam_multiplayer_peer/steam_multiplayer_peer.cpp: In constructor 'godot::SteamMultiplayerPeer::SteamMultiplayerPeer()':
godotsteam/steam_multiplayer_peer/steam_multiplayer_peer.cpp:551:44: error: no matching function for call to 'CCallback<godot::SteamMultiplayerPeer, LobbyChatMsg_t>::CCallback()'
  551 | SteamMultiplayerPeer::SteamMultiplayerPeer()
      |                                            ^

The specific line is this:
STEAM_CALLBACK(SteamMultiplayerPeer, lobby_message_scb, LobbyChatMsg_t, callbackLobbyMessage);

I think it's including the proper library, but I suppose I can't be sure. This is the last of the legitimate errors preventing a build.

Figured it out! Those are function pointers, which are given their value in the initializer. I missed that. I'm real close now, one bug left and it's something I'm too tired to figure out. I'll probably pick it up tomorrow.

Hey, I like the idea for get_direct_messages, but it looks like that's handled mostly by LobbyChatMsg_t callbacks. It looks like it was only partially implemented, as it simply returns an empty array. I may have to dig through GetLobbyChatEntry, after determining the number of available entries, to implement that.
Since the callback should keep everyone synchronized, and I don't see any guarantee that the server will always have a record of every chat message sent, is this basically meant for updating the UI of new lobby members?

I've removed all of the auto statements, as suggested by Godot's style guidelines for Godot 4.0 forks and PRs; and am working on replacing pointers with smart pointers, because we all know we're going to twist them up somewhere. Even more so, I'm not sure I'm going to get another chance to do this for a while.
Godot is running just fine with it, but I haven't actually used it in a project just yet. It's been educational; it's been a little while since I (voluntarily) used C++.
Just a minor update.

@MichaelMacha bless you sir. I've been working in my spare time on using godot steam to set up a MuliplayerPeer client based on cridenour's steam-peer branch. I have been doing it with godot modules as I was unaware of the MultiplayerPeerExtension class in godot4 (not everything in gd4 is fully documented and a little difficult to find at times).
Do you have a public fork for the work you've been doing on this? I'd love to help out if I can and also would like to selfishly base a GDExtension version of what I have so far from your current work haha.

@owenstranathan I will have such a repository available very soon; this began as an experiment with @Gramps repo and many of the suggestions offered here. I want it to be available through GDExtension so that it will integrate nicely with new Godot features, so the end game is hopefully to make it available in the mainline.
It now compiles without so much as a warning, so it's doing precisely what we told it to do... now the question is what that was! I'm going through the functions and documenting them, then, since this amounts to just a handful of additional classes and a mild change to class registration, I'll probably weave them into a fork of Gramp's repo for community testing. I figure if there's anything strange or incomplete, or in violation of C++ standards, I'll see it during the final pass. Then, I'll put it up on Github!
I appreciate your interest! Stay tuned. The only reason it isn't already up and available is the way I got started on this.

OK, I threw the basis of my changes up here! Feedback appreciated but not strictly necessary. I still think I've got some work left to do.
It compiled cleanly with Scons, on QtCreator, on Linux Mint 20.3. for me.

@MichaelMacha this looks very interesting, just starting to dig in this morning. Some questions I have.

How are you building this? I do not see any SConstruct file in the repo
you seem to be including files from godot-cpp  that do not exist in the repo as far as I can tell. Specifically:

#include <godot_cpp/classes/multiplayer_peer_extension.hpp>
#include <godot_cpp/classes/os.hpp>
#include <godot_cpp/classes/time.hpp>
#include <godot_cpp/templates/hash_map.hpp>

those files do not appear to exist at least not in the 4.0 or master branches afaict. Perhaps I'm missing something?
Would love to test building/using this on windows and get this ball rolling :)

@owenstranathan Ah, I should mention the dependencies.
For the godot-cpp stuff, you will want to include the repo next to this repository. Godot-cpp is the language binding to C++. Follow the steps here to use it; you will need to SCons it first. Don't forget to get your extension JSON from the editor first.
As far as how I'm building it, I actually did something a little bit funny here. I cloned the repo into a test Godot project; which is where I incidentally dropped my SConstruct file... which is a little awkward in the context of a Git repository. I'm not entirely sure when that happened.
But to be clear, my repo folder structure is this:
[Godot Project]

godot-cpp
godotsteam
extension_api.json
godotsteam.gdextension
SConstruct

And my SConstruct file, literally, contains this:
#!python
import os

opts = Variables([], ARGUMENTS)

# Gets the standard flags CC, CCX, etc.
env = SConscript("godot-cpp/SConstruct")

# Define our options
opts.Add(PathVariable('target_path', 'The path where the lib is installed.', 'bin/'))
opts.Add(PathVariable('target_name', 'The library name.', 'godotsteam', PathVariable.PathAccept))

# Local dependency paths, adapt them to your setup
steam_lib_path = "godotsteam/sdk/redistributable_bin"

# Updates the environment with the option variables.
opts.Update(env)

# For the reference:
# - CCFLAGS are compilation flags shared between C and C++
# - CFLAGS are for C-specific compilation flags
# - CXXFLAGS are for C++-specific compilation flags
# - CPPFLAGS are for pre-processor flags
# - CPPDEFINES are for pre-processor defines
# - LINKFLAGS are for linking flags

# Check our platform specifics
if env['platform'] == "macos":
    # Set the correct Steam library
    steam_lib_path += "/osx"
    steamworks_library = 'libsteam_api.dylib'

elif env['platform'] in ('linuxbsd', 'linux'):
    # Set correct Steam library
    steam_lib_path += "/linux64"
    steamworks_library = 'libsteam_api.so'

elif env['platform'] == "windows":
    # This makes sure to keep the session environment variables on windows,
    # that way you can run scons in a vs 2017 prompt and it will find all the required tools
    # env.Append(ENV=os.environ)

    # Set correct Steam library
    steam_lib_path += "/win64"
    steamworks_library = 'steam_api64.dll'

# make sure our binding library is properly includes
env.Append(LIBPATH=[steam_lib_path])
env.Append(CPPPATH=['godotsteam/sdk/public'])
env.Append(LIBS=[
    steamworks_library.replace(".dll", "")
])

# tweak this if you want to use different folders, or more folders, to store your source code in.
env.Append(CPPPATH=['godotsteam/'])
sources = Glob('godotsteam/*.cpp')
sources += Glob('godotsteam/steam_multiplayer_peer/*.cpp')

library = env.SharedLibrary(target=env['target_path'] + env['target_name'] + env["suffix"] + env["SHLIBSUFFIX"], source=sources)
env.Depends(library, Command("bin/" + steamworks_library, steam_lib_path + "/" + steamworks_library, Copy("$TARGET", "$SOURCE")))

Default(library)

# Generates help for the -h scons option.
Help(opts.GenerateHelpText(env))

The GDExtension file is no different from what @Gramps is using, just make sure that when you're finished it's pointing to the right location.
In QtCreator, I have the Godot Game folder as my project folder. The build command is scons platform=linux -j8, though you will want to adjust -j to the number of cores you're comfortable using for compilation, and platform it sounds like should be "windows". Clean steps are just scons -c. With that, you can run SCons through Qt Creator with no issues. Don't worry about the <> include syntax, SCons knows how to work with that as long as your Godot-CPP repo is in the expected place.
See, I knew there was an overlooked elephant in the room before I uploaded this, I just wasn't sure what it was and felt like I should show this stuff already. Let me know if you have further issues!

Thanks, yeah I figured out right before you posted that my confusion about the missing includes is because I didn't realize that the majority of the extension api is generated from the extension_api.json

@MichaelMacha that got me a little further, for the sake of not polluting this thread with lots of back and forth. Are you on the CoaguCo discord? We could possibly chat there. I'm seeing you're including some core headers which are not part of the extension api. I'm curious why you're doing this and how you're able to build with it

@owenstranathan I suppose I could look for you there. Could you be more specific about the the headers? To my knowledge, every header I'm using is either C++ Standard, Godot-C++, Steamworks, or Godot Core. Generally this is an amalgamation of a lot of material from other users, though I have cleaned up and standardized it to some extent.

my username on disc is the same as github for easy of use :).
The headers in question are
// Include Godot headers
#include "core/object.h"
#include "scene/resources/texture.h"
#include "core/reference.h"
#include "core/dictionary.h"
#include "core/method_bind_ext.gen.inc"

you're also include some others in other places (this is from godotsteam.h) these are not accessible from gdextension but only when compiling as a godot engine module afaiu
Additionally there are lots of changes I don't exactly understand (though I've not dived very deeply but they do seem to break compilation with my current set up) such as removing use of PackedByteArray and PackedIntArray and replacing it with what appears to be an unimplemented type PoolByteArray and PoolIntArray (perhaps these are types that are in the engine and not in the extension api?) It's not exactly clear to me how you're able to compile a gdextension with the code as it is, though I may still be missing something perhaps? My current guess is that you're somehow compiling with engine source which afaiu is not the intention with gdextension

I haven't touched godotsteam.h; that's straight from Gramps. It compiles for me, so perhaps the problem is elsewhere?

Apparently I forked the wrong version! Here's the correct one, knowing that I specifically made changes to the gdextension branch: https://github.com/MichaelMacha/GodotSteam-MultiplayerPeer

Gonna give that branch a try tonight @MichaelMacha; this seems really neat and a very good leap for the future of networking in Godot steam games

Everybody let me know if they have issues. I haven't integrated it into an actual project yet, there's likely to still be some bumps.

Alas I couldn't get it to work on Godot 4.1, so I'll have to wait a bit to try again

@polyrain GDExtension got some significant updates between 4.0 and 4.1, and only came out the other day. I don't imagine that it's ready-to-go for 4.1 quite yet.

@MichaelMacha I am still running into owenstranathan's issue
// Include Godot headers
#include "core/object.h"
#include "scene/resources/texture.h"
#include "core/reference.h"
#include "core/dictionary.h"
#include "core/method_bind_ext.gen.inc"

I though godot's core stuff wasn't meant to be required, and when I try to compile godot with this as it's module I get .obj and no dlls
I haven't worked with cpp or gdextension before so I'm just a bit confused. I just want to be able to use the SteamMultiplayerPeer

Nevermind I found the right branch, but now I get this error...
link /nologo /WX /OPT:REF /dll /out:bin\godotsteam.windows.template_release.x86_64.dll /implib:bin\godotsteam.windows.template_release.x86_64.lib /LIBPATH:godot-cpp\bin /LIBPATH:godotsteam\sdk\redistributable_bin\win64
libgodot-cpp.windows.template_release.x86_64.lib steam_api64.lib godotsteam\godotsteam.windows.template_release.x86_64.obj godotsteam\register_types.windows.template_release.x86_64.obj
   Creating library bin\godotsteam.windows.template_release.x86_64.lib and object bin\godotsteam.windows.template_release.x86_64.exp
register_types.windows.template_release.x86_64.obj : error LNK2019: unresolved external symbol "public: __cdecl godot::SteamID::SteamID(void)" (??0SteamID@godot@@QEAA@XZ) referenced in function "public: __cdecl SteamConnection::SteamConnection(void)" (??0SteamConnection@@QEAA@XZ)
register_types.windows.template_release.x86_64.obj : error LNK2019: unresolved external symbol "public: static void __cdecl godot::SteamID::_bind_methods(void)" (?_bind_methods@SteamID@godot@@SAXXZ) referenced in function "private: static void __cdecl godot::ClassDB::_register_class<class godot::SteamID,0>(bool)" (??$_register_class@VSteamID@godot@@$0A@@ClassDB@godot@@CAX_N@Z)
register_types.windows.template_release.x86_64.obj : error LNK2019: unresolved external symbol "protected: static void __cdecl SteamConnection::_bind_methods(void)" (?_bind_methods@SteamConnection@@KAXXZ) referenced in
function "private: static void __cdecl godot::ClassDB::_register_class<class SteamConnection,0>(bool)" (??$_register_class@VSteamConnection@@$0A@@ClassDB@godot@@CAX_N@Z)
register_types.windows.template_release.x86_64.obj : error LNK2019: unresolved external symbol "public: virtual __cdecl SteamConnection::~SteamConnection(void)" (??1SteamConnection@@UEAA@XZ) referenced in function "public: virtual void * __cdecl SteamConnection::`scalar deleting destructor'(unsigned int)" (??_GSteamConnection@@UEAAPEAXI@Z)
register_types.windows.template_release.x86_64.obj : error LNK2019: unresolved external symbol "protected: static void __cdecl godot::SteamMultiplayerPeer::_bind_methods(void)" (?_bind_methods@SteamMultiplayerPeer@godot@@KAXXZ) referenced in function "private: static void __cdecl godot::ClassDB::_register_class<class godot::SteamMultiplayerPeer,0>(bool)" (??$_register_class@VSteamMultiplayerPeer@godot@@$0A@@ClassDB@godot@@CAX_N@Z)
register_types.windows.template_release.x86_64.obj : error LNK2019: unresolved external symbol "public: __cdecl godot::SteamMultiplayerPeer::SteamMultiplayerPeer(void)" (??0SteamMultiplayerPeer@godot@@QEAA@XZ) referenced in function "public: static void * __cdecl godot::SteamMultiplayerPeer::create(void *)" (?create@SteamMultiplayerPeer@godot@@SAPEAXPEAX@Z)
bin\godotsteam.windows.template_release.x86_64.dll : fatal error LNK1120: 6 unresolved externals
scons: *** [bin\godotsteam.windows.template_release.x86_64.dll] Error 1120
scons: building terminated because of errors.

@Truc4 You might want to check that branch one more time, "unresolved external symbol" is either a declaration without a definition—and I know that they're defined—or a linker failure. To begin with, make sure you've got the Steamworks Library APIs, of the correct version, in your SDK directory; they do change a little and using the headers with the wrong library can cause this. Also, ensure that you're building the directory that contains things like SteamID.cpp and SteamworksMultiplayerPeer.cpp too.
That said, I generally use GCC on Linux, so I can't reliably test this locally. I can tell you that it almost certainly isn't looking at everything.

@Truc4 I'm not sure what the fork your using has for it's SConstruct file but please make sure it has the lines that include the steam_multiplayer_peer/ folder
gdextension_path = "godotsteam.gdextension"
# tweak this if you want to use different folders, or more folders, to store your source code in.
env.Append(CPPPATH=['godotsteam/'])
sources = Glob('godotsteam/*.cpp')
sources += Glob('godotsteam/steam_multiplayer_peer/*.cpp') # you probably need to add this line

Hope that helps

@owenstranathan Thanks! That was the issue. I can finally move on to something else.

When trying to join a lobby using the SteamMultiplayerPeer I get an error in my Steam.run_callbacks()
W 0:00:06:0059   global.gd:65 @ _process(): Send Error (Unreliable, won't retry): k_EResultFail
  <C++ Source>   godotsteam\steam_multiplayer_peer\steam_connection.cpp:62 @ SteamConnection::send_pending()
  <Stack Trace>  global.gd:65 @ _process()

Can't figure out where to look to find the reason why it's failing, don't know if it's because I set up something wrong or what.
I can never get the lobby_joined signal to fire

Apparently @greenfox1505 wasn't getting notifications!

@Truc4 Thank you for the report, I'll look into it when I'm capable in a few days. It sounds like there may be some code integrated there which improperly carries out the function of send_pending(), or failing that, the code calling it is calling it inappropriately. I have not yet had an opportunity to personally try it in the field, so the information is helpful.

OK gang, the day has come!  @greenfox1505 posted this on the GodotSteam Discord which I wager you all should weigh in on.  The time to merge all the changes together and do something with it is finally upon us, I guess!

Also @greenfox1505 just started up a branch on GodotSteam for this (I assume).  Not 100% sure the plan, maybe he can weigh in on this.

A solution was created for the topic author, details were published on the godot steam server
https://discord.com/channels/564589037411893260/1021131693979090994/1163157677531205707
It's basically the steam sockets solution for gdextension that can be used together with godotsteam.
https://github.com/expressobits/steam-multiplayer-peer
Asset lib link
https://godotengine.org/asset-library/asset/2258

I've tried reading through the thread but can't get a conclusive answer with my limited time, so I was hoping someone knowledgeable can provide a confirmation.
What about this presumably rather common situation:

We already have a game with fully implemented Godot 4 high-level multiplayer API, e.g. ENetMultiplayerPeer.
It does not use a dedicated server; the host is a player.
Now we'd like to allow players to use their Steam friendlist to join games rather than having to go through an archaic join via IP GUI.
Ideally, the hosting player should not be required to forward a port.
The game should not depend on Steam; joining via Steam must remain optional.
Please don't suggest for us to implement the entire netcode a second time with Steam's P2P solution. :)

Can it be done?
I understand that we can create a Steam lobby and use it to exchange IP addresses. Then from there, we could just continue like any non-steam user and call ENetMultiplayerPeer.create_client() to connect to the hosting player. At the very least, this would get rid of the step where the hosting player has to manually share their public IP address, however, it would probably do nothing for not having to go into router settings and configure port forwarding.
Is there a solution that enables P2P games using Godot 4's high-level multiplayer API without requiring players to manually configure port forwarding?

This is beyond the scope of the topic, but answering that this should be an option when hosting the server, whether it should be LAN or Steam, and depending on the option you use ENetPeer or SteamPeer

I've tried reading through the thread but can't get a conclusive answer with my limited time, so I was hoping someone knowledgeable can provide a confirmation.
What about this presumably rather common situation:

We already have a game with fully implemented Godot 4 high-level multiplayer API, e.g. ENetMultiplayerPeer.
It does not use a dedicated server; the host is a player.
Now we'd like to allow players to use their Steam friendlist to join games rather than having to go through an archaic join via IP GUI.
Ideally, the hosting player should not be required to forward a port.
The game should not depend on Steam; joining via Steam must remain optional.
Please don't suggest for us to implement the entire netcode a second time with Steam's P2P solution. :)

Can it be done?
I understand that we can create a Steam lobby and use it to exchange IP addresses. Then from there, we could just continue like any non-steam user and call ENetMultiplayerPeer.create_client() to connect to the hosting player. At the very least, this would get rid of the step where the hosting player has to manually share their public IP address, however, it would probably do nothing for not having to go into router settings and configure port forwarding.
Is there a solution that enables P2P games using Godot 4's high-level multiplayer API without requiring players to manually configure port forwarding?

You're asking a rather big question and a good answer is not possible in this particular forum however I would encourage you to familiarize yourself with the basics of peer to peer (P2P) connectivity via webrtc.
Webrtc is an open standard by the World Wide Web Consortium (W3C) that describes a protocol for establishing a P2P connection between 2 peers on the internet.
There are many articles on the internet that offer primers on this technology, one that I found quickly that seems to hit all the major points can be found here
https://tragofone.com/webrtc-architecture-types-guide/
Godot offers many High level multiplayer (HLM) options as well as the ability to make custom HLM implementations.
They even have a webrtc module you can include in the engine for doing custom P2P via their HLM api.
Many online game engines (Steamworks, Epic Online Services, various first party platforms like PSN, Xbox Live, etc) provide custom systems for establishing a P2P connection. (Note that an online game engine is a set of tools and services for providing online features to games and is distinct for what you would typically call a game engine, i.e. Unreal, Unity, Godot, etc).
Steam has 2 such offerings (which is really just webrtc with different interfaces) called:

SteamNetworkingSockets (https://partner.steamgames.com/doc/api/ISteamNetworkingSockets) - A BSD sockets like interface to creating P2P connections via steam
SteamNetworkingMessages (https://partner.steamgames.com/doc/api/ISteamNetworkingMessages) - A simplified interface to the former

In comparison to the Godot HLM API these are a bit lower level in terms of their controls and the interactions they support.
The base class defining the HLM API in Godot is called MultiplayerPeer and there are various ways to override it and create a custom HLM layer for your game using the P2P technology of your choosing.
See https://docs.godotengine.org/en/stable/classes/class_multiplayerpeerextension.html for one such option using GDExtension in Godot 4.
This issue thread is the initial starting point of an ongoing implementation of HLM in GodotSteam using SteamNetworkingMessages for the connection layer.
It is incomplete and not official afaik and there are multiple implementations floating around from various folks (I am one such folk).
The primary reason it's more difficult than you might imagine to make a P2P connection between 2 computers connected to the internet is because of Network Address Translators (NAT) in modern networking hardware which makes a users public IP ephemeral and unreliable for connecting to their computer specifically. This is the entire purpose of the webrtc RFC.
The webrtc RFC utilizes a signalling server for real time communication between peers without a P2P connection to negotiate the establishment of the P2P connection. viz a way to send arbitrary messages between peers to perform the ICE protocol (https://datatracker.ietf.org/doc/html/rfc8445) which is an integral part of the webrtc specification. Such a signaling service must utilize a third party server there are many options for this but Steam provides one as part of their Steamworks API in whatever technology it is their lobby/messaging/chat backend uses.
Lastly it is not always possible to successfully traverse a NAT and open a P2P connection between 2 peers. Look up NAT types and Symmetric NAT. To work around this the webrtc specification includes a description of a technology called Traversal Using Relays around NAT (TURN). These are servers on the internet that participate in the ICE/STUN process described by W3C and as a fall back provide a relay between 2 peers for sending packets between them. There are some platforms in gaming that dictate that their clients must always use TURN so it's a very necessary part of the whole process.
Steam helpfully provides a TURN solution to games on their platform via Steam Datagram Relays https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay
So if done correctly the aforementioned Steam Networking apis provide a full solution for establishing P2P connectivity between clients seamlessly without any special configuration required by players.
Work to add this to GodotSteam is ongoing.
But to put it very simply. establishing a P2P connection is complicated with modern network topologies and there are numerous technologies available for doing P2P networking most of which leverage webrtc as a fundamental technology.

I've tried reading through the thread but can't get a conclusive answer with my limited time, so I was hoping someone knowledgeable can provide a confirmation.
What about this presumably rather common situation:
* We already have a game with fully implemented Godot 4 high-level multiplayer API, e.g. `ENetMultiplayerPeer`.

* It does not use a dedicated server; the host is a player.

* Now we'd like to allow players to use their Steam friendlist to join games rather than having to go through an archaic join via IP GUI.

* Ideally, the hosting player should not be required to forward a port.

* The game should not depend on Steam; joining via Steam must remain optional.

* Please don't suggest for us to implement the entire netcode a second time with Steam's P2P solution. :)

Can it be done?
I understand that we can create a Steam lobby and use it to exchange IP addresses. Then from there, we could just continue like any non-steam user and call ENetMultiplayerPeer.create_client() to connect to the hosting player. At the very least, this would get rid of the step where the hosting player has to manually share their public IP address, however, it would probably do nothing for not having to go into router settings and configure port forwarding.
Is there a solution that enables P2P games using Godot 4's high-level multiplayer API without requiring players to manually configure port forwarding?

I've had the same scenario for our co-op horror game Disinfection. I've ended up using greenfox1505's multiplayerpeer extension for GodotSteam, which allows you to use Godot's HL networking through Steam relay thus removing the need for NAT punchthrough, servers and all that jazz. And you don't need to change anything in your netcode :)
It's possible to freely switch between ENET's multiplayerpeer and the extension's multiplayerpeer and thus our game supports both playing either through Steam via a list of public lobbies or friend lobbies/invites or through "classic" IP-based P2P (which is also a fallback in case something goes weird with Steam).
Now I will say that greenfox's implementation hasn't been updated in quite a while and a few others are currently working on making an official integration into GodotSteam. So far... it has worked well for Disinfection though and the game has been out since June 2022.

OK, sorry this took me forever but the Multiplayer Peer branch on GodotSteam's repo is now patched and ready for Github Actions to start producing pre-compiles.  I need to go back through the different forks and patch in updates.  Please feel free to submit PR's to this branch to.  I know there were quite a few forks.  If you have some changes / patches squirreled away, let me know.
Gonna start getting the documentation in order today and move some stuff around.
[Initial pre-compiles are here, if you're curious.](https://github.com/CoaguCo-Industries/GodotSteam/releases/tag/v4.5.1-mp

Hey, cool people! There is a new thread for SteamMultiplayerPeer on the Discord where I can hopefully contain most of the information moving forward.
I have gone through all the related threads, this issue, etc. and copied all the information I think is relevant so I can start going over everything and fully get up to speed.
The above thread has links to the multplayer-peer branch in this repo, created by @greenfox1505 and links to @scriptsengineer's repo / Godot Asset Library. I will keep filling in more links and data as I go.
That being said, I will close this thread finally but you can still chat in here if you want!

@EvilDraggie can you please submit or show me some example on how to do it?
Because you said you need no changes in code, I don't understand if you are using rpc, rpc_id... with "remote func" or how you did it?
I am searching for a solution because my game uses a lot of "remote func" and I have no fast solution. Did you found some abstraction for NetworkedMultiplayerENet class? Anway I didn't found anything about this on the documentation of GodotSteam neither the examples. I am using Godot 3.x I cannot change the version because it is a full rewrite of a lot of code, the code project is finished and I prefer to conserve it. It is already tested and working well in LAN or Inet connections.

@Anyeos I'm using the MultiplayerPeer for GodotSteam, there's an official "alpha" now on the Github including an public release, I highly suggest you check that out. I believe there's an example project too.
There also is an official tutorial on the website for compiling if you need to do that.
MultiplayerPeer inherits from the default multiplayer peer of Godot, so the interface and functionality is the same aside from that it uses Steam for the connections/packet transfer.
You can use your RPC's, Synchronizers and Spawners just like you would with ENet. The only difference is how you initialize the peer.
I.e. instead of, for a client:
multiplayer.multiplayer_peer = ENetMultiplayerPeer.new()
multiplayer.multiplayer_peer.create_client(Globals.hostIP, Globals.hostPORT)

etc...
You do:
multiplayer.multiplayer_peer = SteamMultiplayerPeer.new()
multiplayer.multiplayer_peer.as_relay = true
multiplayer.multiplayer_peer.connect_lobby(Globals.SteamLobbyID)

etc...

@EvilDraggie thank you for clarify because it is some confusing what job really do ENet in Godot, As I can see ENet only replaces the last portion of networking, the other things are handled by the inherited class MultiplayerPeer. So as I understand I can use MultiplayerPeer or SteamMultiplayerPeer the same as ENet but without the capabilities of ENet only.
It is good to clarify that on some way to someone like me for save time. if not I already was searching and studying examples but without finding nothing about this. And I can do a test, but if nobody says me it is the correct way I cannot stay in doubt too much time, if I am doing it right or if it is working just by luck.
And to see the Discord link of @Gramps I need first to join the Discord server, so I put the link here to help others found that information too. -> CoaguCo Discord server


Reuse of a rule in one class

It is not possible to use a rule, defined in the parent class of a testclass, twice.
Scenario:
I want to log into a system with two textfields. Both fields should be filled by reusing one rule. Therefore i need two ids, two texts and a rule who can handle two parameters.
When I run the test, the following behavior is observed.
Both texts are written to the last named textfield.

This feature has been implemented in 2.1.0-SNAPSHOT and will be released immediately.


Generate javadoc jars as part of build

Also including fixes to the javadoc to enable the JDK tooling to process our docs (still emits a lot of warnings)
Why:
When we release to maven central if we push up *-javadoc.jar files then we should have javadoc made available for us on https://javadoc.io This would save us having to check in 8MB of generated javadoc HTML somewhere and deal with updating it, we could link to javadoc.io from our asciidoc.

will merge after managed tenants assuming that's headed in soon

@robobario #99 was merged ealier so I'm merging this one.


Find bug during NewsController integration test

When there is no news record in database, the newsList will be a null pointer.

Change the admin 'delete' behavior to user empty list instead of null pointer.


Solve for #44 - pkg_path not exported

Changed Copier processor in Zoom-OutlookPlugin.jss.recipe to use PkgCopier, which exports the pkg_path variable, so it doesn't need to be explicitly defined in child recipes

Thanks!
Sorry for the delay on this.


HashMap.hashCode throws NPE after deserialization

Hi, guys,
There is a bug in the latest release release (2.0.2) regarding HashMap.hashCode. The issue is that after a hash map is deserialized, calling it's hashCode method throws NPE (the field hash is null).
This issue has been fixed on master by a PR made for a refactoring/design improvement. The reason I'm opening this issue is to keep track of this issue on the latest release, and to be able to prioritize the upcoming release accordingly.
While investigating, I realised that the reason behind this problem is that the lazy hash field is marked as transient. While it makes complete sense to throw away evaluated hash value while serialization (as the contents' hash values might change), still, there should be some additional serialization logic (e.g. readObject and writeObject methods) that should reassign the lazy hash field into its unevaluated state. Alternatively, the field hash could be made non-transient and the class javaslang.Lazy could clear its calculated value upon serialization while serializing its supplier. I realize that this is the exact opposite of how Lazy is handling serialization right now (maybe we could make another lazy flavour for this?).
The above PR fixes this issue by removing the lazy hash field altogether.
It might worth mentioning that we stumbled upon this problem after we started using javaslang's persistent collections with Apache Spark, a distributed computing system that heavily relies on serialization of data. Our HashMap instances were put into other hash-based collections, hence the calls to hashCode.

Hi @zsolt-donca,
thank you for using Javaslang and reporting this bug! We will fix it soon in a 2.0.3 release.
Storing the hash lazily for each node of a persistent collections may have a significant impact on memory (additional lazy objects and lambda instances), cpu (instance creation) and garbage collection, depending on the collection size.
Scala and Java do not store the hash, they also compute it every time hashCode is called.
I think most users will not use (large) collections as keys. There is another option (beside making hash lazy in Javaslang):
It might be practicable to create a specific object-wrapper type that lazily provides the hash of the underlying object:
public final class HashKey<K extends Serializable> implements Serializable {

    private static final long serialVersionUID = 1L;

    private final K underlying;
    private transient Lazy<Integer> hash;

    public static <K extends Serializable> HashKey<K> of(K underlying) {
        return new HashKey<>(underlying);
    }

    private HashKey(K underlying) {
        this.underlying = underlying;
        this.hash = Lazy.of(() -> underlying.hashCode());
    }

    public K underlying() {
        return underlying;
    }

    @Override
    public int hashCode() {
        return hash.get();
    }

    @Override
    public boolean equals(Object o) {
        if (o == this) {
            return true;
        } else if (o instanceof HashKey) {
            return underlying.equals(((HashKey) o).underlying());
        } else {
            return false;
        }
    }

    // toString, readObject (initialize hash with a new Lazy), writeObject

}

I'm currently not sure if such a type would fit good into Javaslang. But you might implement your own.
What are your thoughts? Is that an option?

Hi, @danieldietrich ,
Thank you for your quick response!
I see and agree with that storing the hash lazily can have a significant memory overhead. Maybe it would make sense in Scala, a language that has native support for lazy fields, where most of the overhead (the Lazy instance, the lambda for the supplier, the Integer instance, and thus the implications on GC) would not be present. The fact that Scala's immutable collections don't do this is a good sign that, even in Scala, it still doesn't worth it generally.
When the benefits do outweigh the above overhead, such a wrapper class that you proposed seems perfectly feasible to me. We'll consider implementing that class given that our measurements confirm that it worths doing so. As a wise man once said: "Measure, don't premature!"
The fix for this issue in master is perfectly fine by me, and I'm looking forward towards your next release. :) Until then, we'll stick to your snapshot releases.
Keep up the good work!

Hi @zsolt-donca, thank you.
I think the wrapper keys should significantly improve the memory footprint since there is only one wrapper (+ Lazy + Supplier + Integer) for a whole collection instance. If the lazy instance would be baked into the collection, then every value/entry within the collection would have the overhead.
I'm very interested in your insights/measurement. So it would be great if you share it here!
Thx!

Would it be possible to store all the hashCodes in a single static map in Traversable intead?
It could use Guava's cache builder (or anything similar) with weak - System.identityHashCode - keys, and use the iterator() as the base for hashCode calculation.
This way hashCode would be lazy, the memory consumption would be minimal and the speed would be similar to the stored one.

No, please let's follow best practice from other langs like Scala.
The HashKey approach sketched above is O(1) mem and cpu.

So is my suggestion, but for free (i.e. simpler client side code)

I see. Let's not bake it into Javaslang for now.

@paplorinc We should concentrate for now on benchmark coverage of our basic methods (the ones shown in the javadoc performance table). Voxxed gave us the opportunity to write a series of articles on Javaslang.
I'm sure that the benchmarks will be very interesting for the whole Java and Scala community!
This could be a game changer for collection libraries and also for the acceptance of persistent data structures. It would be awesome to see an article here by you!!

I'm sure that the benchmarks will be very interesting for the whole Java and Scala community!

:+1:

Already fixed, needs to be cherry-picked.


[stable/postgresql] Readiness probe "improvements" incompatible with non-default image

Describe the bug
This recent change to the readiness probe assumes that the default Bitnami postgresql image is being used. However, the chart supports overriding the image and the documentation even has an example doing just that. For my use case I am using a custom image which does not have the file /opt/bitnami/postgresql/tmp/.initialized that the readiness probe now checks. As a result, my pod never becomes ready.
Version of Helm and Kubernetes:
Irrelevant
Which chart:
stable/postgresql version 5.2.3
What happened:
Recent readiness probe changes are not compatible with non-bitnami images.
What you expected to happen:
I would expect that documented supported features (using different images) aren't broken by changes to the readiness probe.
How to reproduce it (as minimally and precisely as possible):
Override the default bitnami image with an image that does not create the  /opt/bitnami/postgresql/tmp/.initialized file.
Anything else we need to know:
My current work-around is to use version 5.2.2 of the chart.

Hi @SterlingMcCall,
Thanks for reporting the issue. I was able to reproduce it and I am working on a fix.

@SterlingMcCall The fix is already merged

@tompizmor Thank you for the quick fix! I appreciate it.


Remove last traces of old version scheme

Strip the old PODMAN_VERSION file and its use in the makefile

@rh-atomic-bot r+


Why end-to-end train the GA-RPN with HTC, no improvement compared to the generic HTC

RT,
Why I end-to-end train the GA-RPN with HTC no improvement compared to the generic HTC?
I change the thresholds from 0.5,0.6,0.7 to 0.6,0.7,0.8, but the improvement is almost nothing.
Thanks for your response.

Please use the Reimplementation Questions.


Not able to attack in 1.18.2 only breaking works

HI! Currently, runing last version of Forge, on 1.18.2, and it does the same. I can break block, but can't use any melee attacks. Bow still work.
Thanks for your help!

Fixed in https://github.com/MrCrayfish/Controllable/commit/fe25bba23c15905cca2afe38999f724e255a5500


Library makes no distinction utilities and traversal methods with the same name

There are two separate methods called 'each' with different documentation pages:

https://api.jquery.com/jquery.each/
https://api.jquery.com/each/

The same is true of 'map':

https://api.jquery.com/jQuery.map/
http://api.jquery.com/map/

Right, a single rule covers both cases so all jQuery iteration is deprecated together and can be replaced with native loops and maps.


Remove public Shutdown method on WriteHandler.

We don't ever have long-lived WriteHandler instances, so don't need
external shutdown on them.
Fixes https://github.com/project-chip/connectedhomeip/issues/8651
Problem
Exposed method that can be misused.
Change overview
Stop exposing it.
Testing
Code compiles; there were no consumers of this method outside this class.

We don't ever have long-lived WriteHandler instances, so don't need
external shutdown on them.

I don't understand this.  Write operations take more than zero time (right?),  which is why we need the interaction model to asynchronously track their state in the first place.  So they will be in-flight for some finite amount of time.  How then do these substantively differ from any other type of interaction?

Write operations take more than zero time (right?)

Yes...

which is why we need the interaction model to asynchronously track their state in the first place.

On the sender side.  On the receiver side, the WriteHandler is created, WriteHandler::OnWriteRequest is called, and by the time it returns the WriteHandler has been shut down.  Nothing async going on there.
If we add async behavior under WriteHandler::OnWriteRequest we will obviously need to change things to accommodate that.

How then do these substantively differ from any other type of interaction?

Read and command interactions currently have the ability for the responding end (CommandHandler/ReadHandler) to do async work before responding.  Write interactions do not currently have such an ability: they synchronously respond to the write with a status response.
Again, this is not a feature of the spec, but just of our code.  And if we change this we will want to change how the lifetime of WriteHandler is managed in various ways.  But right now we could literally be allocating our WriteHandler objects on the stack, and we should strongly consider doing that....

Write operations take more than zero time (right?)

Yes...

which is why we need the interaction model to asynchronously track their state in the first place.

On the sender side. On the receiver side, the WriteHandler is created, WriteHandler::OnWriteRequest is called, and by the time it returns the WriteHandler has been shut down. Nothing async going on there.
If we add async behavior under WriteHandler::OnWriteRequest we will obviously need to change things to accommodate that.

How then do these substantively differ from any other type of interaction?

Read and command interactions currently have the ability for the responding end (CommandHandler/ReadHandler) to do async work before responding. Write interactions do not currently have such an ability: they synchronously respond to the write with a status response.
Again, this is not a feature of the spec, but just of our code. And if we change this we will want to change how the lifetime of WriteHandler is managed in various ways. But right now we could literally be allocating our WriteHandler objects on the stack, and we should strongly consider doing that....

OK, this answers my other question about whether writes are sync.  Yes they are, and this approach relies on that.
Also agree that we ought to just being doing this on the stack.


enable statefulset pvc autodeletion feature gate on all clusters

What does this PR do?
Towards https://github.com/giantswarm/giantswarm/issues/29575
This PR enables a feture gate that allows PVCs generated by statefulsets' pods to be automatically deleted when the corresponding pod is deleted. This allows to save on nodes' disk usage as well as preventing potential issues arising from leftover PVCs.
What is the effect of this change to users?
How does it look like?
(Please add anything that represents the change visually. Screenshots, output, logs, ...)
Any background context you can provide?
(Please link public issues or summarize if not public.)
What is needed from the reviewers?
Do the docs need to be updated?
Should this change be mentioned in the release notes?

[x] CHANGELOG.md has been updated (if it exists)

Eh, I think you pretty much destroyed the values.schema.json unintentionally. 🤣

Try implementing your changes to the default values in the default section of the values.schema.json and run make generate afterwards. That should do the trick. :)

@Gacko It should be all good now :)
But I must say that generating the values file from the schema is quite tiresome.

@Gacko I think I addressed all your concerns :)


CVC check doesn't work against a validmastercard

Used my mastercard to test, has a cvc w 2 digits over 5

The CVC of MasterCard always have a length of 3 digits. For example: 123
What is the length of your CVC?

3 digits total

It should work fine. The CVC is only validated against the length defined in the rules.
This code will output bool(true):
var_dump(CreditCard::validCvc('123', 'mastercard'));

Check that you CVC don't contain spaces with the trim function

Verified that I'm running trim  "179" fails still


Return the names of the detected SSIDs

It would be nice if ESPWifiLocation could return to the sketch the names of the detected SSIDs. My sketch, for example, needs to check for the presence of a certain SSID. Of course I could do just another scan in my sketch, but that is wasteful in terms of energy consumption and time needed since ESPWifiLocation is doing a scan already.

You can use getSurroundingWiFiJson() bit internally it scans networks.
You may use WiFi.SSID(index) to get last scan result.

Thank you. Maybe this could be documented/shown in an example sketch.


Updating the Json Engine to be less dependency heavy.

Fix for #5.

Updated Json Engine throughout to JsonCpp to loosen dependence tree.
Updated CMake for OSX and Linux.
Still needs update for Windows Visual Studio project.

@ryanbartley hi ， I use your fork under mac os,  and compiled , when after cmake ..,  i type in make, then It has two error below:
[ 72%] Building CXX object IO-glTF/CMakeFiles/IO-glTF.dir/gltfWriter-Texture.cpp.o [ 75%] Building CXX object IO-glTF/CMakeFiles/IO-glTF.dir/gltfWriter.cpp.o /Users/adams/Documents/code/FBX-glTF/IO-glTF/gltfWriter.cpp:433:20: error:        cannot increment value of type 'std::initializer_list<int>'                                 sIdentity4Mat[i++] = val;                                               ~^ /Users/adams/Documents/code/FBX-glTF/IO-glTF/gltfWriter.cpp:597:8: error:        cannot increment value of type 'std::initializer_list<int>'                         ar[k++] = ((r [i] [j])) ;                            ~^ 2 errors generated. make[2]: *** [IO-glTF/CMakeFiles/IO-glTF.dir/gltfWriter.cpp.o] Error 1 make[1]: *** [IO-glTF/CMakeFiles/IO-glTF.dir/all] Error 2 make: *** [all] Error 2
I'm not family with c++, do you know how to solve this problem?

I had problems compiling due to the Casablanca dependency, this patch applied cleanly and I was able to convert a model with it.  +1 for this patch!


Remove deprecated functions

They seem to interfere with documentation creation as the parameters do now show up correctly here: https://py.api.tudat.space/en/latest/aerodynamic_coefficients.html#tudatpy.numerical_simulation.environment_setup.aerodynamic_coefficients.constant
The functions for which this happens aligns exactly with the functions that have been deprecated in the code below.

I've changed the target of the merge to the develop branch. There, I think these deprecated functions can now be safely removed.


[BOT PR] to update dependencies of istio

This PR will be merged automatically once checks are successful.
none

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: istio-testing
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: rshriram
If they are not already assigned, you can assign the PR to them by writing /assign @rshriram in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@istio-testing: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
76684c4d34a4fac97261047c2178d9db0be24a15
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
76684c4d34a4fac97261047c2178d9db0be24a15
link
/test istio-unit-tests

prow/istio-pilot-e2e-envoyv2-v1alpha3.sh
76684c4d34a4fac97261047c2178d9db0be24a15
link
/test istio-pilot-e2e-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Feature/138 add possibility to replace queryset filter with custom function

Resolves https://github.com/wagtail/wagtail-autocomplete/issues/138

Thanks, @simon-retsch! We'll take a look at this soon

Nice, thanks, if there is anything I can help with or if something is unclear about the PR, please let me know.


Add support images -a

Signed-off-by: Xian Chaobo<EMAIL_ADDRESS>

@jimmyxian It seems we need some integration tests for this PR :)

@chanwit Yeah..Forget to git commit. :)

@chanwit https://github.com/docker/swarm/pull/1085#issuecomment-126522219

all=true : get all images(including intermediate layers)
all=false : get the images which have repotag

@jimmyxian Thank for clarifying.
LGTM

LGTM


update docs

Hi, please update docs to avoid confusion,
since nette 2.2 registration of helper shoud be done like this (or may be you know better)
    protected function createTemplate($class = null) {
        $template = parent::createTemplate($class);
        $template->setTranslator($this->translator);
        return $template;
    }

Hi and thank you.
The thing is that it's been a while since I last used my own translator, so there was no reason for me to update it.
This is how kdyby/translation does it (here is code)
So for us it should be something like that:
$helpers = $translator->createTemplateHelpers();

$template = parent::createTemplate($class);
$template->getLatte()->addFilter('getTranslator', [$helpers, 'getTranslator']);
$template->getLatte()->addFilter('translate', [$helpers, 'translate']);

return $template;

Which is definitelly not so nice..... Your code would also work but without "debug mode".


Update c-ares to 1.16.0

Could you please update c-ares to 1.16.0.
We had a problem with sporadic stalls (6 seconds) tracing back to this c-ares issue which is solved in 1.16.0.
PS: We use a workaround at the moment, by using the native resolver and "options single-request" in resolv.conf as suggested in DNS NSS improvement by Ulrich Drepper.

We had a problem with sporadic stalls (6 seconds) tracing back to this c-ares issue which is solved in 1.16.0.

It is probably a good idea to try do an upgrade in any case, but for this specific problem, are you sure that you are hitting the c-ares bug described in https://github.com/c-ares/c-ares/issues/285, and not something else?
I ask because grpc library doesn't use ares_getaddrinfo, and instead uses the ares_gethostbyname API. And when doing this, the A and AAAA records are queried in separate, parallel calls to ares_gethotbyname.

Thanks for the quick reply @apolcyn !
Maybe you are right and we pinned it on the wrong issue, i will check our backtrace, when i am in the office again...
Fact is that the stall always occurred when the non-existing AAAA record returned first, never when the A record returned first. Using the same gRPC version (1.24.2) and only exchanging c-ares we could not re-produce the stalls anymore. But they always were elusive...

 We just got another stall with our patched version. :(

And when doing this, the A and AAAA records are queried in separate, parallel calls to ares_gethostbyname.

@apoclyn I suppose there is no way to force gRPC to do these calls in sequence instead of parallel?

@apoclyn I suppose there is no way to force gRPC to do these calls in sequence instead of parallel?

Correct, there's not

Okay, thanks for the information.
Whats the best way to proceed?
Close this issue and file a feature request to get something like resolv.conf's 'options single-request' in grpc?

PS: We use a workaround at the moment, by using the native resolver and "options single-request" in resolv.conf as suggested in DNS NSS improvement by Ulrich Drepper.

This seems like a reasonable workaround. Though the "native" resolver isn't the default, it is supported and a valid option.
Getting something like the "single-request" option to work with the c-ares resolver could be possible, but I suspect would require some amount of surgery to the c-ares library (or alternatively at a higher layer in the gRPC library). It's possible that the c-ares library does actually support something like this and I'm not aware of it, in which case it would likely be somewhat straightforward to do this. I don't think we'll have time to get to this in the near future though, but we could potentially accept a patch if someone had one. The relevant code where gRPC integrates with c-ares starts in https://github.com/grpc/grpc/blob/master/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc.
All of that said, it not done already it might be worth verifying that the problem is actually the DNS server being overloaded by capturing the DNS traffic with e.g. tcpdump - if you have a pcap file, we could take a look.


[Feature]: show usage stats during trial as well for business edition

Is there an existing issue for this?

[X] I have searched the existing issues

Summary
Customers while upgrading from a trial license to paid license business edition might want to get an estimate of their usage and billing amount. This has also been requested by one of customers on trial - link to conversation here
Why should this be worked on?
This will give customers an idea of their bill once they upgrade to paid edition, and hence making it easier for them to upgrade and also reduce the likelihood of churn once they upgrade

We are also evaluating the paid plan and would like to know how to monitor the usage by our user. We're on a budget and would like to get an estimate of costs.
Is there a table to query to get those statistics?

Hi @umbearto, while we don't have a way to show it directly in the product today, could you share the email through which you signed up for trial of business plan? You can share it here or send it to me on<EMAIL_ADDRESS>and I can help with usage stats


Bug

Version
3.0.9 (4d8d5d6)
Seed
R720-0VKA-ZJNH70MZ9Y00HL27BVR4VYZZFY3XVZZZKPY
Scenario
RE 2 (Claire A)
Player Character
Piers
What happened?
No way to progress, first three areas are linear and they lead to a save point with no further exit.

This seed is for Claire B not Claire A.

This seed is for Claire B not Claire A.

Regardless, the bug still persists in all three cases. I won't reply to all three reports.

When you generate a seed you must only play on the scenario it was generated for. This seed was for Claire B, so you can not play Claire A, it must be Claire B.

When you generate a seed you must only play on the scenario it was generated for. This seed was for Claire B, so you can not play Claire A, it must be Claire B.

The penny dropped for me afterwards, thanks for clarifying, my apologies! Thanks for replying :)


Session not cleared if user uses Back after logout

Checklist

[ ] The issue can be reproduced in the auth0-react sample app (or N/A).
[X] I have looked into the Readme, Examples, and FAQ and have not found a suitable solution or answer.
[X] I have looked into the API documentation and have not found a suitable solution or answer.
[X] I have searched the issues and have not found a suitable solution or answer.
[X] I have searched the Auth0 Community forums and have not found a suitable solution or answer.
[X] I agree to the terms within the Auth0 Code of Conduct.

Description
After calling logout the user is redirect back to our login page, but if they go back with their browser they can get back into the application and perform authenticated actions. Calling refresh at any time throws them out of the application, and the auth0 logs show a successful logout.
To add more mystery to this, it does not happen consistently. For example, if the developer tools are open in the browser, the problem goes away and going back throws the user onto the login screen.
Reproduction

login
logout
press back
find yourself logged in again.

It seems to be mitigated by having the developer tools open, but is pretty much consistent in normal use. Also unable to replicate it on a localhost dev build.
Additional context
This is our auth provider
  const onRedirectCallback = (appState?: AppState) => {
    navigate(appState?.returnTo || window.location.pathname);
  };
  ...
      <Auth0Provider
        domain={config.auth0.domain}
        clientId={config.auth0.clientId}
        authorizationParams={{
          redirect_uri: window.location.origin,
          audience: config.auth0.audience,
        }}
        onRedirectCallback={onRedirectCallback}
        useRefreshTokens
        cacheLocation={config.auth0.cacheLocation} // undefined expect for e2e tests
      >

auth0-react version
v2.2.1
React version
17.0.2
Which browsers have you tested in?
Chrome

Hi @KlausVii - thanks for raising this
This is a performance feature of Chrome (and Safari & FF) called Back/forward cache
When you click the Back button, Chrome loads the previous page seemingly in an instant, because the entire page is restored from memory - there's not much the SDK can do, but you can disable bfcache on your browser if you don't want the feature.

Thanks for sharing that link @KlausVii - you can certainly do that on your application if you want.
I can't think of a good reason why we wouldn't add something like that to the Spa SDK (at least behind a config option) - let me discuss it with the team.

Hi @KlausVii - I'm just doing some more investigation into this

login
logout
press back
find yourself logged in again.

Are you able to share a link where I can reproduce this behaviour?
Also, did you try adding the event listener  and did it resolve your issue?

@adamjmcgrath Sorry our application is not open to the public, so cannot really share a link.
I have not tried the event listener as of now, we parked this kind of hoping your fix would resolve it 😅
I'll let you know if/when I find time to test this.

np, thanks @KlausVii
I only ask because I'm investigating this and can't reproduce it on localhost or another domain and I'm wondering how bfcache works with redirects (cross domain or otherwise)
If you test it lmk if it works also, if you can share a HAR file (with secrets redacted) that demonstrates the issue I'd also be interested in looking at it.

Also looking to get this addressed. The event listener workaround is not working for me as the cookie still seems to exist on back tap, or at least in the time that the code checks it. I even tried an interval to check it.
Alternatively we always reload on pageshow persisted=true check but that is a burden on the user.

Thanks everyone for the patience here.
For now, we have been unable to reproduce this, but believe we have a bit of missing information.
Can someone provide some information for us to help reproduce this, in order for us to consider how to address this?

Are you using auth0-react in your application?
Are you using auth0.js in your application?
Are you using lock in your application?

To avoid any confusion, it would help if someone could share an exact reproduction of a client application that allows to reproduce the behavior.
@zebapy  I am asking the above because I noticed the information you shared with our team through different channels indicates the use of auth0.js (e.g. it's set as the Auth0Client header on the call to /authorize), and you mention lock. Which I am trying to understand how they fit in the story here.
Additionally, I think there may be other factors at play here that are important to get some information on:

Are you using the new or classic login experience on Auth0?
In case you are using the classic login experience, which login template are you using if you are using?

Hi @frederikprijck ,
Im having similiar issues.
Currently using: @auth0/auth0-react": "2.2.4".
Not using auth0.js.
Not using lock
Currently this only occurs in Firefox and only with the built app, it does not occur when the dev server is running. After logging out, once Auth0 has redirected back to our app, if the user clicks the back button it looks like it is loading the page from cache (I assume this is the bfcache as I dont see any network requests other than icons in the network tab). I also noticed that here:
https://github.com/auth0/auth0-react/blob/main/src/auth0-provider.tsx#L202
It only dispatches the "LOGOUT" action if you specify an openURL, ie. if you let auth0 do the redirect back to the client app it doesnt dispatch the action and and doesnt set isAutheticated to false. I assume this is done as its going to load a new copy of the client after the redirect anyway where the isAuthenticated status will be setup fresh anyway. But if the user clicks the back button and the browser uses the bfcache then the page will be repopulated with the Auth0Provider still having isAuthenticated set to true. The app is then accessible for a bit, I assume until something checks the authenticated cookies which were correctly removed on logout.
In browsers where it doesn't rerender the pages on clicking back like chrome I see that on clicking back I get the logout url appearing in the network tab so Im assuming it isnt loading the page from the bfcache.
Im probably barking up the wrong tree but would it help if the client dispatched the logout action even if the user hasn't provided an openUrl in the logout options?
Alternatively is there something that is meant to prevent the browser using the bfcache when clicking back after logout?
Thanks for your time
Michael


ART-2945: {runtime_assembly} variable substitution

If assemblies are enabled, replace {runtime_assembly} in
ocp-build-data with the runtime assembly name.

Codecov Report

Merging #422 (2425656) into master (d0244e1) will increase coverage by 31.18%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           master     #422       +/-   ##
===========================================
+ Coverage        0   31.18%   +31.18%
===========================================
  Files           0       37       +37
  Lines           0     8439     +8439
  Branches        0     1693     +1693
===========================================
+ Hits            0     2632     +2632
- Misses          0     5655     +5655
- Partials        0      152      +152

Impacted Files
Coverage Δ

doozerlib/runtime.py
27.02% <0.00%> (ø)

doozerlib/rpmcfg.py
52.63% <0.00%> (ø)

doozerlib/brew.py
37.77% <0.00%> (ø)

doozerlib/distgit.py
31.93% <0.00%> (ø)

doozerlib/gitdata.py
20.83% <0.00%> (ø)

doozerlib/source_modifications.py
65.16% <0.00%> (ø)

doozerlib/__init__.py
54.54% <0.00%> (ø)

doozerlib/dotconfig.py
21.42% <0.00%> (ø)

doozerlib/cli/images_health.py
62.02% <0.00%> (ø)

doozerlib/cli/cli_opts.py
81.25% <0.00%> (ø)

... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d0244e1...2425656. Read the comment docs.


Remove deprecated brush_shape

🧰 Task
Following on from #2426 by @ppwadhwa, are we now ready to remove the deprecated brush shape and only allow for the square brush? It will simplify the codebase. @jni if you're good for this to go now then @ppwadhwa you could pick this up when you have time - should be pretty easy once green light from @jni. We'd just be left with the circle brush as the only brush type after this.

Yup I'm good with this! 👍

@ppwadhwa this could be a good thing to pick up during the "quality sprint" if you like too


Unable to save projection when using select all Projection Fields

The issue: UI (OK button) unresponsive when configuring a simple projection and using select all Projection Fields. When selecting fields one by one, using chrome, OK button is responsive.
Using Firefox, the OK button is never responsive.

@shahsank3t , can you take a look ? This appears to be the code in latest master.

@arunmahadevan sure I'll take a look into it soon

@sameer79 thanks for addressing the issue, merged PR to master.


[CT-3173] [Bug] Postgres + Grants + Username-With-Dashes does not work

Is this a new bug in dbt-core?

[ ] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
When I configure grants in dbt for a postgres database and a db role like 'foo-bar' I get postgres syntax errors because the name is not quoted.
If I try to quote it like "\"foo-bar\"" it works for the grants part but breaks in the incremental grants part. Looks like some places do not strip and some places do strip those extra quotes I added to my db role name.
Expected Behavior
I should not have to put in extra quotes - dbt should be able to create the appropriate database syntax for supported role names.
Steps To Reproduce
Create a postgres role with dashes in the name.
Try to have dbt assign grants to that user - it will break.
Add manual quotes - it will work.
Try to make a incremental model with those same grants - it will break.
Relevant log output
No response
Environment
- OS: macOS ventura 13.6
- Python: 3.10.12
- dbt: 1.6.2

Which database adapter are you using with dbt?
postgres
Additional Context
No response

As an update and workaround to this issue - I have been able to leverage the post-hook feature of models to apply the grants myself - where I just make sure I properly quote my db role name.

Thanks for reporting this @tomans-spirent !
Could you try this workaround and see if it also works for you?

Seems like a similar issue - but no such luck if I add that revoke macro to my project.
Some extra context from my dbt run log output:
Database Error in model my_model (models/my_group/my_model.sql)
  syntax error at or near "-"
  LINE 4: ...db"."schema"."my_model" to grantee-with-dash;

With the following in my dbt_project.yml
models:
  my_group:
    +grants:
      select: ["grantee-with-dash"]

Ah, sorry I only gave you the revoke-side of the story @tomans-spirent ! 😅
Related issues
It looks to me like there are two different (but very related) things going on:

dbt-postgres/redshift doesn't quote username when granting (#8751)
dbt-postgres/redshift doesn't quote username when revoking (#6444)

See below for two different approaches you can try.
Option 1: Add more quotes
Does it work if you add double quotes within your configuration?
For example, like this for models YAML configuration within dbt_project.yml (note both single and double quotes):
models:
  my_group:
    +grants:
      select: ['"grantee-with-dash"']

If this does work for you, I think you'll still need to that macro you just added to handle the revoke side of things.
Option 2: Add a grant-related macro
https://github.com/dbt-labs/dbt-redshift/pull/282 has a proposed solution for both of those issues listed above (which is converted from being redshift-specific to default below):
{%- macro default__get_grant_sql(relation, privilege, grantees) -%}
    grant {{ privilege }} on {{ relation }} to
    {% for grantee in grantees %}
    {{ adapter.quote(grantee) }}
    {% if not loop.last %},{% endif %}
    {% endfor %}
{%- endmacro -%}

{%- macro default__get_revoke_sql(relation, privilege, grantees) -%}
    revoke {{ privilege }} on {{ relation }} from
    {% for grantee in grantees %}
    {{ adapter.quote(grantee) }}
    {% if not loop.last %},{% endif %}
    {% endfor %}
{%- endmacro -%}

Trade-offs
I didn't try these out myself, but I'm assuming it requires all grants to be case-sensitive when they are written in YAML and config for dbt models. So that trade-off would probably work for you, but might break others if we were to just roll these macros out everywhere as-is.
Wanna give one or both of these ideas a try and let us know how it goes and what you think?

First option:
works EXCEPT for incremental table grants.
Second option:
Works for views, materialized, and incremental tables!

I have added a file: macros/github-issue-8751.sql
With the contents from your comment and it patches the issue for me locally

Interesting point re: the case sensitivity and existing backwards compatibility.
I tested against postgres and indeed it was case sensitive.
So: I can verify the second workaround works with the caveats you expected.

Thanks for trying out both of those options with a host of types (view, table, & incremental) @tomans-spirent ! 🏆
I'm going to label this as Refinement for us to decide whether to roll-out the second option as the default or not. One thing for us to consider is if we can/should add a quote_grants config for grants (or something else similar to quote_columns). If we add some kind of quoting-related config, hopefully it would fit together with https://github.com/dbt-labs/dbt-core/issues/2986.