Change the way posts data are stored

Context
During the same conversation already mentioned inside #60 that I had with @angelorc yesterday, another possible problem that he raised about our chain is regarding the storage of posts.
He told me that in Bitsong they have been testing through all the year different methods of storing songs data and metadata so that the chain can always perform good without becoming larger than it should be. He told me that the best way they found out in the end was storing all the metadata on IPFS and linking the CID (the IPFS hash of the file containing such data) on the chain itself.
After such conversation, I've done some hypotheses on how large our chain could become if we keep all the posts contents on-chain. This is what I got.
Chain space analysis
Premises
All the following analysis have been done referring to the JSON format of posts, which is the one used when exporting them inside the genesis or importing them during an upgrade process. This has been done due to the fact that is simpler to verify how large a JSON file is instead of how much space it occupies inside KV stores.
Current status
Currently Posts are stored using the following JSON structure:
{
  "id": "134",
  "parent_id": "156",
  "message": "This is an estimated average long post message",
  "created": "566879",
  "last_edited": "899978225",
  "allows_comments": false,
  "subspace": "dwitter",
  "optional_data": {
    "local_id": "e3b98a4da31a127d4bde6e43033f66ba274cab0eb7eb1c70ec41402bf6273dd8"
  },
  "owner": "desmos1wjtg20d7hl9y409hhfydeqaph5pnfmzxlgxjg0"
}

In this case I've put an average length value for each field, considering the optional_data value to have only one key inside, which should be the average case. This JSON has a size on disk of 378 bytes.
Now, with the introduction of media-supporting posts (#36), an average JSON file of such time could look like the following:
{
  "id": "134",
  "parent_id": "156",
  "message": "This is an estimated average long post message",
  "created": "566879",
  "last_edited": "899978225",
  "allows_comments": false,
  "subspace": "dwitter",
  "optional_data": {
    "local_id": "e3b98a4da31a127d4bde6e43033f66ba274cab0eb7eb1c70ec41402bf6273dd8"
  },
  "owner": "desmos1wjtg20d7hl9y409hhfydeqaph5pnfmzxlgxjg0",
  "medias": [
    {
      "uri": "http://ipfs.pics/QmXr1iejP2zHptFFDr3hycZvbaXaQNwrK6VVXYbxFAYQ7x",
      "mime_type": "image/png"
    },
    {
      "uri": "http://ipfs.pics/QmTbx9HLaN7gsKiunNc5NWvNRytydKn5uWWmpUzAHPU3NS",
      "mime_type": "image/png"
    },
    {
      "uri": "http://ipfs.pics/QmQZWZxfSHB3FpU8w4EMfX9bF52wUyPCiVNjtjf6jeNBBp",
      "mime_type": "image/png"
    }
  ]
}

Due to the higher amount of information stored, such JSON has an on-disk size of 768 bytes.
Let's now consider the case in which Desmos gains 1/1000th of Twitter usage. This could lead to 2 millions posts per each (source of Twitter usage stats: InternetLiveStats).
Considering an average of 1/5 posts being a media post, this would lead to 400.000 media posts/year and 1.600.000 text posts/year created. Considering 378 bytes per each text post and 768 bytes per each media post, this would sum up to approximately 0.92GB of posts per year.
If we scale up to 1/100th of Twitter usage, we then would collect 9.2GB/year of only posts. This amount is crazily high and I personally think that we should reduce it as in the long run it might become a problem considering that there are a lot of posts type we still need to define (e.g. #14).
Solution
The solution of storing all the post content into an IPFS file could be a good way to decrease the size of the chain on-disk. What we could do is define a new Post structure that is stored like the following:
{
  "id": "87556",
  "created": "566879",
  "content": "QmP8jTG1m9GSDJLCbeWhVSVgEzCPPwXRdCRuJtQ5Tz9Kc9",
  "owner": "desmos1wjtg20d7hl9y409hhfydeqaph5pnfmzxlgxjg0"
}

Inside the IPFS file reachable using the content reference, we can then have different JSON structures based on the type of the post itself. As an example, we could have
Text post
{
  "type": "post/Text",
  "parent_id": "156",
  "message": "This is an estimated average long post message",
  "allows_comments": false,
  "subspace": "dwitter",
  "optional_data": {
    "local_id": "e3b98a4da31a127d4bde6e43033f66ba274cab0eb7eb1c70ec41402bf6273dd8"
  }
}

Media post
{
  "type": "post/Media",
  "parent_id": "156",
  "message": "This is an estimated average long post message",
  "allows_comments": false,
  "subspace": "dwitter",
  "optional_data": {
    "local_id": "e3b98a4da31a127d4bde6e43033f66ba274cab0eb7eb1c70ec41402bf6273dd8"
  },
  "medias": [
    {
      "uri": "http://ipfs.pics/QmXr1iejP2zHptFFDr3hycZvbaXaQNwrK6VVXYbxFAYQ7x",
      "mime_type": "image/png"
    },
    {
      "uri": "http://ipfs.pics/QmTbx9HLaN7gsKiunNc5NWvNRytydKn5uWWmpUzAHPU3NS",
      "mime_type": "image/png"
    },
    {
      "uri": "http://ipfs.pics/QmQZWZxfSHB3FpU8w4EMfX9bF52wUyPCiVNjtjf6jeNBBp",
      "mime_type": "image/png"
    }
  ]
}

Of course this are just generic examples and better schemes should be defined more in depth and maybe event stored on chain for a generic reference.
On-disk space changes
Thanks to this approach, all different posts would have an on-disk size of just 166 bytes, which would reduce the above yearly disk-space increase from 0.92GB/year to 0.332GB/year (−69,456%) considering 2.000.000 posts/year.
The best thing is that this approach keeps a linear increase of the disk space that is not based on the posts contents at all. If all users suddenly started creating media posts, on chain they would weight the same as text posts so it's also a spam prevention system.
Querying
Thank to the system proposed inside #60, the parser could simply ready the posts from the chain and then get the real content from IPFS, index it and putting into the database allowing for easier queries.
Consideration on IPFS
IPFS has proven to be reliable and we can even improve its availability by creating a cluster or a private network so that posts content will be always available, although I don't think this will be needed.
Cons of this
The only con of this is that the chain would not be able to check the contents of the posts during the transaction processing. However, I don't know if that might be a problem for us or not.
Also, clients would have to upload the content before the transaction, so we are moving onto them this responsibility. This can however be fixed by creating a REST API that performs such operation for the client.
Conclusion
I personally think this should be the way to follow that would allow us to:

focus less on the development of the posts module.
Once you define the specification for each post, you do not need to edit the way they are stored on chain.

achieve a higher scalability of the chain itself.
As posts are lighter on disk, the chain can scale better and transactions will be processed faster.

I would love to hear @kwunyeung and @bragaz feedback on this and what it might be improved or should change on such approach.

I am against this approach.
I agree the binary media data should be on an external storage service like IPFS but not the content of the post and the metadata. They should be on the chain as they are the result of consensus and the values of the users would like to store under their accounts. If we only store the CID of the IPFS, it's lack of sovereignty and ownership of the original content creator. Do you mean the chain software should cater the file upload to IPFS? Do the node operators need to maintain IPFS nodes and pay the fee for the entire process? Unless we have another layer in the protocol which is for storing the content only in the system like data replicator in Solana, provider in Akash and storage node in Oasis, I do not agree on this approach. I don't see that text-based storage is a severe issue. Yes maybe in a long run the storage can be 3 times larger than just storing CID but solely because of reducing storage size while losing sovereignty is not what I would expect. Some longer terms solution could be data compression in KVstore (I'm not sure if Tendermint is already doing this as I know goleveldb can have data compression), or have some old data in archive mode.
We can take Steem as an example. Here is one of their data stored on chain.
https://steemd.com/fiction/@stormlight24/change-of-tide-university-path-short-fantasy-story-part-20
There are a lot of data but still, they are text only (markdown). As long as the data is storing in binary and compressed, it won't be too heavy I think.

Do you mean the chain software should cater the file upload to IPFS? Do the node operators need to maintain IPFS nodes and pay the fee for the entire process?

I don't think that's what Riccardo meant. He propose to move this responsibility to the clients,
they would manage the upload to IPFS.

Some longer terms solution could be data compression in KVstore (I'm not sure if Tendermint is already doing this as I know goleveldb can have data compression), or have some old data in archive mode.

What do you mean with archive mode? Compress them after some time they were on chain?
I agree that media data should be cater on IPFS, I really don't know what will happen when we would have a huge amount of posts.
Will these data slow the system dramatically? And if they would cause such a problem, how will we able to face them? By Change the whole system? It would be crazy.
For now I would probably agree with the IPFS solution for both media and text posts to ensure a fast service in the long term.
I remember once while talking with Alessio from cosmos that he warned us to not put much data on chain and that it would be better for us to keep the chain as light as possibile, but this is only his opinion, we could ask to anyone from cosmos/tendermint team if they are thinking about that.

What do you mean with archive mode? Compress them after some time they were on chain?

Archived the block data so they won't be accessible directly. Maybe this is not eligible in our case.

I remember once while talking with Alessio from cosmos that he warned us to not put much data on chain and that it would be better for us to keep the chain as light as possibile

I agree with this. I think there are different scenarios. For a tweet-like message, Facebook-like post, they should be stored on chain. For a Medium-like blog, the full text body storing on IPFS and only store the metadata on chain is a better solution. This matches the case of storing photos and musics. For example, if I want to build an IG-like app, I would store the photos on IPFS and only store the description in the post and like it to the CID of the photo.
Maybe we should think about limiting the length of text in a post. This helps limiting the size of an tx an gas use. And then we build some demos on how Desmos can work with IPFS to build some interesting applications.

For a tweet-like message, Facebook-like post, they should be stored on chain. For a Medium-like blog, the full text body storing on IPFS and only store the metadata on chain is a better solution. This matches the case of storing photos and musics. For example, if I want to build an IG-like app, I would store the photos on IPFS and only store the description in the post and like it to the CID of the photo.

I think it's a good idea, it avoids chain's pollution and in the meantime it let tracks of user's posts on chain and preserve sovereignty.

Maybe we should think about limiting the length of text in a post. This helps limiting the size of an tx an gas use. And then we build some demos on how Desmos can work with IPFS to build some interesting applications.

Yes we could do that, sounds good to me!
What do @RiccardoM you think about all this?

I personally think that this solution might be the best one:

Maybe we should think about limiting the length of text in a post.

I go as far as to say that the maximum post length should be something like 500 characters, which is 1.78 times a Tweet length.

And then we build some demos on how Desmos can work with IPFS to build some interesting applications.

This can be done inside Dwitter too, which can use IPFS for posts longer than 500 characters.

I'm ok for 500 characters. Does it matter if it has to be a number which is a power of 2? As the limit of a memo in a tx is 256.
I think Dwitter (Mooncake) should not go for the IPFS long post. That should be another dapp which is aimed for blog posting. We should keep the function of each dapp focus to a single problem.

I'm ok for 500 characters. Does it matter if it has to be a number which is a power of 2? As the limit of a memo in a tx is 256.

No, it doesn't really matter. We can fix any length we want :smile:

I think Dwitter (Mooncake) should not go for the IPFS long post. That should be another dapp which is aimed for blog posting. We should keep the function of each dapp focus to a single problem.

Ok perfect!

Closing this in favor of #67


Delete data directory

Do we need the data directory since we have CIL-data?

We need to add the cil data package to the dependency and actually create the cil-data package first!

We need to add the cil data package to the dependency and actually create the cil-data package first!


Feat(deps): bulk updates

Description
Music year: 1955

Updated numerous packages
Applied linting updates
Husky formatting updates

Checklist

[x] Added or updated tests where appropriate
[x] Does this PR update a package with a breaking change

:tada: This PR is included in version 1.609.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


fix: Fix this[writeSym] is not a function error

This error is caused by passing the Pino logger to the GitHub octokit library.
More details here: https://github.com/pinojs/pino/issues/545

:tada: This PR is included in version 3.11.10 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


ceph-mgr questions

fatal: [master-01]: FAILED! => {"changed": false, "msg": "Unable to start service ceph-mgr@master-01: Job for<EMAIL_ADDRESS>failed because the control process exited with error code.\nSee "systemctl status<EMAIL_ADDRESS>and "journalctl -xe" for details.\n"}
fatal: [master-02]: FAILED! => {"changed": false, "msg": "Unable to start service ceph-mgr@master-02: Job for<EMAIL_ADDRESS>failed because the control process exited with error code.\nSee "systemctl status<EMAIL_ADDRESS>and "journalctl -xe" for details.\n"}
fatal: [master-03]: FAILED! => {"changed": false, "msg": "Unable to start service ceph-mgr@master-03: Job for<EMAIL_ADDRESS>failed because the control process exited with error code.\nSee "systemctl status<EMAIL_ADDRESS>and "journalctl -xe" for details.\n"}

could you share a bit more details?
inventory host file ? group_vars/* ? ansible version ? OS ? systemctl status ? /var/log/ceph/*.log ?
i'm not sure how to help here otherwise

I'm wondering if this might be related to the @czunker's issue in #2768 but without more details it's hard to help.
@ww110052181 please provide sudo journalctl -x -e -l -u ceph-mgr@master-XX output

@guits  thx a lot,close this issue


Command pfctl does not seems to be executed

Hello Derek,
I guess I have a correct config file, but command pfctl doesn't seems to get executed properly.
I've got the following logs
Sep  8 20:17:37 vm-haproxy-1 hfm[6708]: '<IP_ADDRESS>/haproxy2' run 1a24e52cc9c1958:<IP_ADDRESS>/haproxy2:1 change command produced output: {[47 115 98 105 110 47 112 102 99 116 108 32 45 113 32 45 116 32 104 97 112 114 111 120 121 45 105 110 116 45 115 116 100 32 45 84 32 100 101 108 101 116 101 32 49 57 50 46 49 54 56 46 51 46 50 52 52 47 51 50 10] 0 0}
Could you give me advise to check why ?
Regards
Thomas

Anywhere you can post your config?

Hello,
Before your answer I was thinking about right, but I still got the same issue with sudo.
You'll find my config file below

interval = 10s
<IP_ADDRESS> {
status = "enabled"
interval = 1s
# matches test interval
# interval_fail =
haproxy1 {
interval = 200ms
interval_fail = 10s
test = "fetch"
test_arguments = [
"-o",
"/dev/null",
"-T1",
"-q",
"http://<IP_ADDRESS>:8404/stats"
]
change_fail = "/bin/sh"
change_fail_arguments = [ "-c", "'/usr/local/bin/sudo /sbin/pfctl -q -t haproxy-int-std -T delete <IP_ADDRESS>/32'" ]
change_success = "/bin/sh"
change_success_arguments = [ "-c", "'/usr/local/bin/sudo /sbin/pfctl -q -t haproxy-int-std -T add <IP_ADDRESS>/32'" ]
}
}
<IP_ADDRESS> {
status = "enabled"
interval = 1s
# matches test interval
# interval_fail =
haproxy2 {
interval = 200ms
interval_fail = 10s
test = "fetch"
test_arguments = [
"-o",
"/dev/null",
"-T1",
"-q",
"http://<IP_ADDRESS>:8404/stats"
]
change_fail = "/bin/sh"
change_fail_arguments = [ "-c", "'usr/local/bin/sudo /sbin/pfctl -q -t haproxy-int-std -T delete <IP_ADDRESS>/32'" ]
change_success = "/bin/sh"
change_success_arguments = [ "-c", "'/usr/local/bin/sudo /sbin/pfctl -q -t haproxy-int-std -T add <IP_ADDRESS>/32'" ]
}
}

Hello,
Sorry, I haven't search properly, It works.
Many thanks for your time
Regards
Thomas

Thanks!  For what it's worth, you're looking at the output of the command, encoded as decimal ASCII-value.
Also, I haven't run this thing in production - because of #1 .  It looks like it was resolved in FreeBSD but haven't had time to test it out.  Also, check out the comments here:
https://derek.chezmarcotte.ca/haproxy-carp-on-freebsd#comment-524117


Support other reporters

I can see that only the dot-reporter is supported. I had issues with CSI ansi codes when I wrote my mocha-mode as well. It's pretty easy to get around that though by something like this:
(save-excursion
    (replace-regexp "\\[[0-9]+[a-z]" "" nil (point-min) (point-max)))

(edit the post to see the exact code)
Creating this issue here as a reminder.

Closing via https://github.com/scottaj/mocha.el/pull/31


Fix a method missing exception due to version difference

Fixes #667
@EricMSFT @MrayerMSFT @jinglou- @AndreSayreMSFT

Added a quick fix for another bug since it deals with the same method. Code looks good to me, but suggestion from @EricMSFT is not without merit.


simple covering method that doesn't require the response consumer

I'm not sure why we need both the consumer and the response returned, but this method would simplify the usages that just want the result, without requiring an empty consumer.

Makes sense.


[New chain]: Japan Open Chain Mainnet

Summary
I would like to add v1.4.1 on Japan Open Chain Mainnet
Chain ID
81
Chain detail URL
https://chainlist.org/chain/81
RPC URL
https://rpc-1.japanopenchain.org:8545
Blockscout Client URL
No response
Etherscan Client URL
No response
Etherscan Client API URL
No response
Version
1.4.1
Address (Master copy)
0x41675C099F32341bf84BFc5382aF534df5C7461a
Deployment Tx hash (Master copy)
0x6419837814fc8ddb23a4f668af7627f7d1172a162a80f8d4c0966d0b9e36771d
Block explorer URL (Master copy)
https://explorer.japanopenchain.org/address/0x41675C099F32341bf84BFc5382aF534df5C7461a
Address (Master copy L2)
0x29fcB43b46531BcA003ddC8FCB67FFE91900C762
Deployment Tx hash (Master copy L2)
0x0dafdcca85491507a39f83c27ac5f5b0ca06ff1c6b7dda02357fce8df688e0bf
Block explorer URL (Master copy L2)
https://explorer.japanopenchain.org/address/0x29fcB43b46531BcA003ddC8FCB67FFE91900C762
Address (Proxy factory)
0x4e1DCf7AD4e460CfD30791CCC4F9c8a4f820ec67
Deployment Tx hash (Proxy factory)
0x879f0d351f716d8a665ed25b969df578890a7af6ec67961778c59216f3e94465
Block explorer URL (Proxy factory)
https://explorer.japanopenchain.org/address/0x4e1DCf7AD4e460CfD30791CCC4F9c8a4f820ec67

/execute


Need a thumbnail

This repository need to have a thumbnail or table to add all projects that have been completed

Noted!


Merge multiple DT projects

As a user of DT, i would like to provide sbomasm with multiple project ids and it should merge it and output it either to a file or to another DT project.

/assign


Possible Errors with Xidi

Hi Samuel,
Hope things are well! The other week we released an update to the Silent Hill 2: Enhanced Edition project which includes the debut of Xidi with it. We've had some reports of the project being unable to launch due to VC Redist packages needing to be installed, along with others having an "error 0xc0000142" code while Xidi was in the game's directory.
Elisha made a build of Xidi with all dependencies removed but users are still reporting the error mentioned above. May we ask for your thoughts on this matter? Please see the tickets below for some of the user reports:

https://github.com/elishacloud/Silent-Hill-2-Enhancements/issues/732
https://github.com/elishacloud/Silent-Hill-2-Enhancements/issues/729
https://github.com/elishacloud/Silent-Hill-2-Enhancements/issues/736
https://github.com/elishacloud/Silent-Hill-2-Enhancements/issues/725 (this is for Linux/Steam Deck, but might be related to the issue at hand?)
https://github.com/elishacloud/Silent-Hill-2-Enhancements/issues/740

Thank you!

Based on Elisha's latest custom build seeming to help and some testing I did on my own, I think there are a couple of separate issues taking place here at the same time.
It seems like the fix that would work for your users is to disable enhanced processor instructions and change the dependency to use static linking. I already refactored all the compiler settings so that these settings can be changed by modifying Properties/BuildSettings.props, which is much less likely to be modified than the individual Visual Studio project files themselves. The idea is to make it far easier and more maintainable to create a custom build and keep it updated with newer versions of Xidi.
More details below.

External dependency on Visual Studio 2022 runtime

I know that in the past this was fixable by building Xidi with the runtime linked statically. I do not own a Steam Deck myself but I have seen other users saying Xidi works on Steam Deck by installing the runtime and setting it up correctly. A custom build that statically links in Visual Studio runtimes (as Elisha has already been doing) would very likely work too.

Startup error 0xc0000142 and enhanced processor instructions

This particular error code translates to STATUS_DLL_INIT_FAILED and indicates the DLL is crashing during initialization.
I was able to reproduce the 0xc0000142 error by adding this code into DllMain and making a custom build:
*((int*)0) = 55;

This code attempts to write to a null pointer, which causes a crash. I also tried this:
__asm {
   ud2
}

This code causes the compiler to emit an instruction that is invalid on every x86 processor. It also leads to a crash.
Since it seems that removing the enhanced processor instructions compiler setting fixes the problem, probably what is happening is some instruction that was introduced in AVX-generation processors (~2011) happens to be present in my official builds and is getting executed when the application loads Xidi, hence the crash. I gather that this is happening on Windows 11 for ARM, which suggests that whatever instruction happens to be causing this problem is one that the x86-to-ARM translator doesn't currently support.

I will certainly consider turning off the enhanced instructions in a future release. The original rationale for enabling the enhanced processor instructions was to cause the compiler to generate the most efficient code it could for things like bit manipulation, floating-point math (force feedback uses this extensively), and so on without having to worry about the constraints of very old processors. However, given that ARM is as modern as it gets for Windows 11, and the x86 emulator doesn't support those enhanced instructions, I suppose I might need to re-think that line of reasoning. Furthermore, to be fair, I also haven't measured the performance impact of using the enhanced instructions.
As for the use of /MT vs /MD, there are a few reasons why I prefer /MD for the official builds. More on this below.
One option you could consider if you want to avoid having to rebuild Xidi with /MT is to add the individual runtime DLL files it needs to your Xidi installation package and distribute them alongside Xidi. This is presumably as simple as adding a few extra binaries into the package like msvcp140.dll and msvcp140_atomic_wait.dll, all of which are installed along with Visual Studio. (I haven't tested this myself, but I think it might work.) It appears that Microsoft explicitly permits this, and you would be able to distribute the official builds of Xidi without worrying about the external dependency.
Reason 1: Licensing
I have actually tried to figure out if Microsoft allows binaries to be distributed with its copyrighted runtime code embedded within, and I haven't found a clear answer.
Reason 2: Redundancy
This likely doesn't apply to your project, but I fully intend Xidi to be loaded into programs and games alongside other DLLs of mine that also use the Visual C++ runtime. This already happens whenever the HookModule method is used to force a game to load Xidi because Hookshot also uses the Visual C++ runtime, and I have a few future projects in mind that would add even more such loaded modules. There is no good reason to have multiple complete copies of the same runtime loaded into memory - it just seems wasteful of resources.
Reason 3: Update Management
Users of Xidi, myself included, probably have multiple games installed on their machines all of which use Xidi in one way or another. Let's suppose there is a security vulnerability or other important update Microsoft releases to fix issues with their Visual C++ runtime. If I were to use /MT, I would need to rebuild Xidi, and users would subsequently need to replace all of their installed instances with the updated version. On the other hand, if I continue using /MD, I can just rely on Microsoft to push out an update without the users needing to do anything.

I will certainly consider turning off the enhanced instructions in a future release. The original rationale for enabling the enhanced processor instructions was to cause the compiler to generate the most efficient code it could for things like bit manipulation, floating-point math (force feedback uses this extensively), and so on without having to worry about the constraints of very old processors. However, given that ARM is as modern as it gets for Windows 11, and the x86 emulator doesn't support those enhanced instructions, I suppose I might need to re-think that line of reasoning. Furthermore, to be fair, I also haven't measured the performance impact of using the enhanced instructions.
As for the use of /MT vs /MD, there are a few reasons why I prefer /MD for the official builds. More on this below.
One option you could consider if you want to avoid having to rebuild Xidi with /MT is to add the individual runtime DLL files it needs to your Xidi installation package and distribute them alongside Xidi. This is presumably as simple as adding a few extra binaries into the package like msvcp140.dll and msvcp140_atomic_wait.dll, all of which are installed along with Visual Studio. (I haven't tested this myself, but I think it might work.) It appears that Microsoft explicitly permits this, and you would be able to distribute the official builds of Xidi without worrying about the external dependency.
Reason 1: Licensing
I have actually tried to figure out if Microsoft allows binaries to be distributed with its copyrighted runtime code embedded within, and I haven't found a clear answer.
Reason 2: Redundancy
This likely doesn't apply to your project, but I fully intend Xidi to be loaded into programs and games alongside other DLLs of mine that also use the Visual C++ runtime. This already happens whenever the HookModule method is used to force a game to load Xidi because Hookshot also uses the Visual C++ runtime, and I have a few future projects in mind that would add even more such loaded modules. There is no good reason to have multiple complete copies of the same runtime loaded into memory - it just seems wasteful of resources.
Reason 3: Update Management
Users of Xidi, myself included, probably have multiple games installed on their machines all of which use Xidi in one way or another. Let's suppose there is a security vulnerability or other important update Microsoft releases to fix issues with their Visual C++ runtime. If I were to use /MT, I would need to rebuild Xidi, and users would subsequently need to replace all of their installed instances with the updated version. On the other hand, if I continue using /MD, I can just rely on Microsoft to push out an update without the users needing to do anything.

Imo I think the installer should just do an OS check and install both X64 and X86 redists and call it a day.

I will certainly consider turning off the enhanced instructions in a future release.

Ok, great.  Thanks!

This is presumably as simple as adding a few extra binaries into the package like msvcp140.dll and msvcp140_atomic_wait.dll, all of which are installed along with Visual Studio.

The issue with just including these dlls is that when you open the file you can see other dll files besides just these ones that are statically linked that may cause other issues.  In other words, we may not always be able to just add those few dlls and be done with it.  See graphic below when compiling with /MD.

In my experience, relying on all these other dlls has a chance of causing issues if one of those dlls has an issue, or is missing, which I have seen.  All of these dlls tend to be flaky, in my experience, in Windows.  Having less things you need to rely on is better, in my opinion.
However, all of these are gone and only a few core Windows dll files are linked when you compile with /MT so there is a much better chance of correct functionality.

I have actually tried to figure out if Microsoft allows binaries to be distributed with its copyrighted runtime code embedded within, and I haven't found a clear answer.

Based on the documentation (here, here and here) it is pretty clear, to me at least, that we can distribute these files as we chose, including embedding them in the code.

There is no good reason to have multiple complete copies of the same runtime loaded into memory - it just seems wasteful of resources.

Given how small these embedded files are and how much memory modern computers have, I doubt this is an issue.  However, besides all that, only a part of the dlls that you embed are included, not the whole dll.
When you compile with /MT it only increases the size of Xidi by only about 120 KBs.  However, msvcp140.dll is 437 KBs by itself.  And then msvcp140_atomic_wait.dll is another 46 KBs.  It would take four other dlls with embedded runtime files to just equal these two, and that does not include all the other dlls that need to be loaded that I showed you above.
In that vast majority of cases embedding the files into the dll will decrease the memory usage, not increase it.

Users of Xidi, myself included, probably have multiple games installed on their machines all of which use Xidi in one way or another.  Let's suppose there is a security vulnerability or other important update Microsoft releases to fix issues with their Visual C++ runtime.

Keep in mind that there could be a vulnerability in Xidi that would cause a user to update all their instances, so that may require users to update all their copies of Xidi regardless of the runtimes.  Also, loading the full runtime increases the risk since all the functions are there for an attacker to use.
However, when you embed the runtime into Xidi then it reduces the risk in two ways:

The embedded functions are in a different location for different programs.  For instance, they are in a different location in Xidi then they are in the sh2-enhce project I'm working.  This makes it much harder for an attacker to use since they would likely need to make a specific attack for Xidi and then a different specific attack for sh2-enhce, rather than just a generic attack on the runtime.
Not all of the runtime functions are embedded so there is a good chance that the vulnerable ones won't even be included so there is no vulnerability at all in many cases.

Anyways, this whole discussion is a philosophical discussion.  For me, I would rather increase the chance of my code working and decrease the amount of work that my users need to do to get the software working by embedding the runtime into the project.  And since I don't really see much negatives to it, only positives, that is the solution I think is best.

I definitely agree that much of the discussion here is philosophical. Our projects have different needs and we both have differing views on the subject, so I'll start with the practical before moving onto the more philosophical.
Practical
While I do acknowledge and respect that your use case might call for /MT, I don't intend to change Xidi's official builds to /MT from /MD. Our differing views on this subject is actually one of the main motivations for refactoring the build settings into a separate file that is unlikely to be changed. That way, if you decide to maintain a fork that uses /MT, your maintenance burden will be almost nothing.
Philosophical
The suggestion I made last time regarding distributing the runtime DLLs alongside Xidi falls under the local deployment method, which Microsoft lists in one of the links you shared but doesn't recommend over using the redist installer. It would probably work, though I haven't tested it myself.

The issue with just including these dlls is that when you open the file you can see other dll files besides just these ones that are statically linked that may cause other issues.

DLLs of the form "api-ms-win-*.dll" are actually just "umbrella libraries" that are not expected to exist as literal files. This falls under the Windows feature known as "api sets." As an example, based on this feature, Xidi's hook module for WinMM checks for the joystick API set rather than winmm.dll explicitly. If it gets a handle back and you were to call GetModuleFileName() on that handle, chances are you would get back the path to the system's winmm.dll file.

I would rather increase the chance of my code working and decrease the amount of work that my users need to do to get the software working by embedding the runtime into the project.

I haven't heard of any issues from the users of Xidi that installing the runtime library redist has been an issue or even an annoyance. For as long as Xidi has been released it has been a documented requirement, and users both on and off of Github are very familiar with the fact that they need to do the installation first. Users have even published guides on how to get the redist working on Steam Deck running Linux.
Other applications that are distributed via installers often will install the redist silently by running the redist installer executable with the right command-line parameters, which requires no work on the part of users. It happens in the background (maybe with a UAC prompt if that's needed) and then it's done.
All of the above together tells me that installing the redist is an easy thing for users to do, and it's a once-and-done operation. Furthermore, Microsoft itself actually recommends linking dynamically with their runtime instead of statically. This is explicitly stated in one of the links you shared.
I'll revisit the remaining discussion points in the same order as with the last post: licensing, redundancy, and update management.
Licensing
I read through the three links you shared, and they all talk about distributing the DLLs or the redist installer EXEs unmodified. I haven't seen anything explicit about embedding bits and pieces of their copyrighted binary code into a DLL and distributing that. Static linking isn't just a matter of embedding an entire DLL file, it means the linker picks and chooses which bits of their binary to include, which one could argue counts as a "modification."
In any case I tend to be very risk-averse when it comes to licensing. If I'm not explicitly told I can do something with somebody else's code then I assume I can't do it. That's just my own personal view on the subject, though. Maybe Microsoft is totally okay with their runtime code being statically linked and distributed as such.
Redundancy

Given how small these embedded files are and how much memory modern computers have, I doubt this is an issue.
...
In that vast majority of cases embedding the files into the dll will decrease the memory usage, not increase it.

I recognize that the total binary size growth is quite small when switching to /MT, especially compared to the size of the DLLs themselves. But when I say "redundancy" I am not concerned so much with binary size as I am with runtime behavior.
Let's think back to the use case I mentioned earlier of having 5 DLLs loaded into the same process, all linked with the same Visual C++ runtime library. This is a realistic use case given some of the ideas I have for future projects that I intend to work together.
If we link with /MT then each DLL has to do a separate runtime initialization of all of the common runtime features. Furthermore, since memory allocation and even the entire heap and its associated management is implemented in the runtime, we would get at least 5 separately-managed heaps being initialized and managed, one for each DLL. (Operators new and delete, smart pointers like std::unique_ptr, and so on all make use of the runtime's heap memory management by default.) While it is true that each DLL will only contain the exact parts of the runtime that it needs, there is still the runtime core that every binary needs.
If we link with /MD then we do all of the above steps once, and we also guarantee that all of the loaded DLLs get the exact same version of the runtime. We only have one heap, we only need one copy of whatever common data structures the runtime needs to support its functionality, and we do all of the initialization only once.
Update Management

Keep in mind that there could be a vulnerability in Xidi that would cause a user to update all their instances, so that may require users to update all their copies of Xidi regardless of the runtimes.

True, and an update will be required no matter what if there is a security vulnerability in a project or even just a new feature that users want to use. But we need to consider which code is more likely to have a bug, and what the possible severity of that bug might be.
Xidi is completely implemented in user-mode using documented standard interfaces. It is a relatively small project that serves one specific, relatively niche, purpose.
The Visual C++ runtime, on the other hand, implements an entire general-purpose standard that supports applications running in a wide variety of environments, including production. It has a much larger surface area and orders of magnitude more users. Furthermore, Microsoft has complete freedom to implement it as they see fit, using interfaces and techniques that I would never dream of using because they aren't necessarily publicly available. (I'm not saying they do this, but they could.)

Also, loading the full runtime increases the risk since all the functions are there for an attacker to use.

Also true, but there is still the runtime core that gets included with statically-linked modules. Furthermore, I actually have no idea which parts of the runtime the linker chooses to include in my binary. If Xidi were programmed to use a particular part of the runtime, Microsoft's implementation could very well include a dependency on another part of the runtime, and short of doing a lot of digging into binary dumps, I would have no easy way of being able to tell. In other words, it is not immediately obvious which parts of the runtime get included in a statically-linked binary and which are left out. The problem is I would have to know that in order to determine whether or not an update to the runtime necessitates an update to Xidi. This is not true if I keep using /MD - Microsoft takes care of all the updates, and neither I nor the users of Xidi have to worry about it at all.

Thanks for the comments @samuelgr.  I respect your thoughts and what you're doing for your project is probably the right thing!  👍
Another way to solve this is for us to include Xidi project as a submodule of sh2-enhce.  This would eliminate all the issues here, as I would compile Xidi inside sh2-enhce like I do with d3d8to9 and other modules.
Anyways, just a couple of random thoughts below:

Our differing views on this subject is actually one of the main motivations for refactoring the build settings into a separate file that is unlikely to be changed. That way, if you decide to maintain a fork that uses /MT, your maintenance burden will be almost nothing.

Yes, I like this idea and it works for us if we decided to recompile the build each new release.

I haven't heard of any issues from the users of Xidi that installing the runtime library redist has been an issue or even an annoyance.

I believe the users for sh2-enhce are different than the users for Xidi.  Xidi is already a technical product for a technical user.  sh2-enhce is for any user, even non-technical ones.  We have a lot of issues with users installing (or not installing the correct runtimes).  We want to reduce the number of trouble tickets we have here.

Other applications that are distributed via installers often will install the redist silently by running the redist installer executable with the right command-line parameters

Frankly, I don't like doing this because it seems like one more moving part to the install that can go wrong (and I have seen the runtime fail to install before).  I like the KISS principle.  Besides all that, it causes issues with uninstall.  How do you know when to uninstall it?  If you remove it and other programs are relying on it then you break those.  If you don't remove it then you break the main thesis of uninstall, which is to remove everything that you added.  I really don't like leaving things around on my user's computers.

The suggestion I made last time regarding distributing the runtime DLLs alongside Xidi falls under the local deployment method .... It would probably work, though I haven't tested it myself.

Yes, maybe this is the way to go.  If we can only include a couple of extra files for the runtimes then it solves some of the install and uninstall issues.  I was just concerned if this would work or if there are other dependencies besides just the runtimes.  I need to test this more, but it may save me from having to recompile Xidi for each release.

Also true, but there is still the runtime core that gets included with statically-linked modules. Furthermore, I actually have no idea which parts of the runtime the linker chooses to include in my binary.

That is kind of the point.  Unless there is a specific attack on Xidi then there is no need for an update, generic attacks on the runtime won't work against Xidi if the runtime is embedded.

I read through the three links you shared, and they all talk about distributing the DLLs or the redist installer EXEs unmodified.

You're not modifying the exe files or any of the dll files.
Nowhere in the licensing does it say you cannot include these embedded.  The licensing is what restricts you and if it doesn't restrict you then it doesn't restrict you.
This is a built-in option that Microsoft gives for released builds from the very tool that they created.  This article here even tells you how to build "a release version of your project" by "static linking" the runtimes using the "/MT" command.  If they didn't want you to release code using this option then why did they give you the ability to do this with release builds?

There is no way it would stand up in court that you did anything wrong if you embedded the functionality with the /MT command.

With commit 0ad5223188d6dd9b28cfc77c6ed5ee047bfc2425 I've removed the enhanced processor instructions from the build settings.

Thanks @samuelgr!


Default memory and cpu limit to that of Node Capacity

Current memory and cpu limits for containers are set to a random number which does not signify the logic limits applied by the kernel. We can instead default cpu and memory limits to that of the node.
cc @piosz @mwielgus

For reference, the memory limits are currently being set to -1 and the cpu limits are being set to 99 millicores.

Is this in Heapster?
On Fri, Sep 18, 2015 at 2:24 PM, Alexandros Mavrogiannis <
<EMAIL_ADDRESS>wrote:

For reference, the memory limits are currently being set to -1 and the cpu
limits are being set to 99 millicores.
—
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/heapster/issues/581#issuecomment-141572184
.

Can you provide more context? I don't understand how this issue is related to Heapster.

This is defined in CAdvisor.
Machine containers are populated here: https://github.com/google/cadvisor/blob/master/container/raw/handler.go#L168
The behavior you're describing could be handled here as well

@mwielgus: If limits are not set on containers, the node's capacity is the limit. So instead of defaulting memory limit to some arbitrary high value, we can set it to node capacity. Similarly for cpu, we can set it to cpu capacity. This defaulting behavior is only required if one of the sources doesn't do it automatically. For example, Cadvisor could perform this defaulting.

Obsolete.

Has this been fixed for real? This will affect monitoring of best effort
pods...
On Wed, Mar 2, 2016 at 7:45 AM, Marcin Wielgus<EMAIL_ADDRESS>wrote:

Closed #581 https://github.com/kubernetes/heapster/issues/581.
—
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/heapster/issues/581#event-574843464.

As for now we aggregate limit/request metric for node. It's no longer a machine capacity.


⚠ ClamAV has degraded performance

In 63508a1, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 195 ms

Resolved: ClamAV performance has improved in 991f254.


Please enable Tute annotation access via Google Could Storage

Hi Nicole,
Thank you for adding this feature, but I'm getting the following access error:
$ gsutil ls gs://tute_db
AccessDeniedException: 403 Forbidden
$

This is based on the following provided link:
https://docs.google.com/document/d/1_Kryc4qAqw1NRezaqDJ1tXUSCbxEkKK4SSi_kZuyHtU/pub
Thanks,
Paul

Hi Paul,
This very cool resource is provided by Tute Genomics, so please be sure to send any requests or feedback about it to<EMAIL_ADDRESS>Thanks!
Nicole

Aha!  Will do :)
Thanks Nicole,
Paul


[Question] How to make joins

Is there a possibility to make joins using Speedment?
The documentation says that there is no need for it and I can do it like that:
// Find the owner of the orange carrot
Optional hare = carrots.stream()
.filter(Carrot.NAME.equal("Orange"))
.map(hares.finderBy(Carrot.OWNER))
.findAny();"
But is it possible to do something like this?
SELECT  *
FROM    Player p
WHERE   p.Name = 'Godlex'
AND     EXISTS (SELECT 1 FROM User u WHERE u.Name = p.VisibleName AND u.Public = true)

There is a join example [here] (https://github.com/speedment/speedment#join) and in your specific case you could do something like this:
Stream<Player> result = players.stream()
    .filter(Player.NAME.equal("Godlex"))
    .filter(p -> users.stream()
        .filter(User.NAME.equal(p.getVisibleName))
        .filter(User.PUBLIC.equal(true))
        .findAny()
        .isPresent()
    );


chore: update to Artifact Registry

@averikitsch since I made a change, please check over then feel free to merge.


Fix various incompatible pointer types.

This patch fixes a bunch of type errors like the following, by casting the expected types, working around gcc warnings that have gone fatal in gcc 14:
src/index.c:1263:59: error: passing argument 2 of ‘ti_querys_2d_multi’ from incompatible pointer type
+[-Wincompatible-pointer-types]
1263 |          sequential_iter_t *siter = ti_querys_2d_multi(t, regions, n_sub_list);
     |                                                           ^~~~~~~
     |                                                           |
     |                                                           char **
src/pairix.h:131:73: note: expected ‘const char **’ but argument is of type ‘char **’

Note the pointer returned by ti_iter_first cast to "int" in function ti_query_2d_tid, in src/index.c, feels like an incident waiting to happen, but the function is expected to return addresses or error codes, so it is left as-is due to lack of better idea for the moment.

can confirm this solution works for me @clarabakker


Change th name!

Hi, how can I change the name dislayed in  element?
Because I cannot make a json whit blank spaces.
There is a solution?

Hi @Vincenzo1997, please see example below - you can use arbitrary string values for the JSON object keys, which flow through to the .  Please let me know if it helps.
import SearchTable from 'reactable-search';
import ReactDOM from 'react-dom';

const rows = [
  {'foo bar': 1, 'baz qux': 2},
  {'foo bar': 2, 'baz qux': 3},
];

ReactDOM.render(
  <SearchTable
    searchPrompt="Type to search"
    rows={rows}/>,
  document.getElementByID("root");


Advice on adding encryption support

Hi. First of all, apologies if opening an issue isn't the best way to start this dialogue. I tried posting to the Google group, but it appears that I don't have permissions to start a thread there.
I would like to add support for symmetric encryption to PynamoDB. What I have in mind would be similar to the Java library for encrypting data in DynamoDB (https://github.com/awslabs/aws-dynamodb-encryption-java). I'd like to get others' thoughts on how to implement this feature--especially the thoughts of those who would be approving my pull request.
I don't know this code base well at all, as I've been using boto3, but here's what I'm thinking after reading some of the code:

It looks like the _serialize and from_raw_data functions in models.py are the places to do the encryption and decryption, respectively. I see two possible approaches to controlling whether a Model encrypts its data. The approach that I'm leaning toward is to give the base Model class a member that determines whether it encrypts its data and to examine that member in _serialize and from_raw_data. The other approach that comes to mind is to define an EncryptedModel class that inherits from Model and overrides those two functions.

There needs to be a way to make certain attributes within a Model non-encrypted. I think that a good way to meet this requirement is to add a "do_not_encrypt"  field to the Attribute class.

I'm not sure what the best way to specify where a Model gets its encryption key(s) from is, but one possibility is to give the Model class an "encryption_key_source" member that refers to some object that produces the necessary information. The common use case might involve creating a single encryption key source and pointing all of one's Model subclasses to it.

Please let me know what you think.

I decided not to go down this path after doing more research.  I do believe that adding encryption in the manner described above would not be too difficult; however, I fear that ensuring compatibility with the Java encryption library, such that one could access the same encrypted tables in Java and Python, would be significantly more difficult. The ability to access tables with Java is necessary as long as Java is the only language with support for transactions in DynamoDB. Rather than implement both encryption and transactions in Python, I decided to rewrite my backend in Java.


Feature Request: add redirectUrl field in OAuth2AuthConfig for authWithOAuth2

In the new Simplified OAuth2 authentication flow(#55) there is no option to specify a custom redirect url after the flow is completed
this will be useful in apps where deep linking is setup  and pocketbase backend is hosted on a different url

This doesn't make much sense to me. The idea of the "all-in-one" call is to handle everything in the user app without redirect.
If you want to provide a custom redirect url and to handle a deeplink, you can use directly the "Manual code exchange" flow, aka. the "old" way (internally the "all-in-one" call is using it).

If I have

a SvelteKit application (using Pocketbase) on mydomain.com  (vercel)
a PB instance running on https://mypbinstance.fly.dev (fly.io)

does this men that I cannot use the Simplified OAuth2 authentication flow. Is that correct, or have I misunderstood?

@einarpersson The "all-in-one" OAuth2 flow works only in environments where you can open a browser window (eg. in a client-side browser, cli or desktop application).
It won't work in a node server-side context (unless you proxy it somehow) because it requires a long running persistent HTTP connection to remain active during the entire OAuth2 signin/signup process.

@einarpersson it will work in your mentioned scenario the problem I had with it in my use case was that my svelte app is running in an android app so the user is not being automatically redirected to my app after authentication flow is completed just a go back message was being displayed


session not created: Chrome version must be between 70 and 73

Testbench Version: 4.4.1
Laravel Version: 6.x
PHP Version: 7.3
Database Driver & Version: n/a

Description:
1) Orchestra\Testbench\Dusk\Tests\Browser\AuthTest::can_authenticate_user
Facebook\WebDriver\Exception\SessionNotCreatedException: session not created: Chrome version must be between 70 and 73
  (Driver info: chromedriver=2.45.615355 (d5698f682d8b2742017df6c81e0bd8e6a3063189),platform=Mac OS X 10.14.6 x86_64)

/Users/miki/Projects/orchestra/testbench-dusk/vendor/facebook/webdriver/lib/Exception/WebDriverException.php:154
/Users/miki/Projects/orchestra/testbench-dusk/vendor/facebook/webdriver/lib/Remote/HttpCommandExecutor.php:331
/Users/miki/Projects/orchestra/testbench-dusk/vendor/facebook/webdriver/lib/Remote/RemoteWebDriver.php:140
/Users/miki/Projects/orchestra/testbench-dusk/src/TestCase.php:106
/Users/miki/Projects/orchestra/testbench-dusk/vendor/laravel/dusk/src/Concerns/ProvidesBrowser.php:192
/Users/miki/Projects/orchestra/testbench-dusk/vendor/laravel/framework/src/Illuminate/Support/helpers.php:393
/Users/miki/Projects/orchestra/testbench-dusk/vendor/laravel/dusk/src/Concerns/ProvidesBrowser.php:193
/Users/miki/Projects/orchestra/testbench-dusk/vendor/laravel/dusk/src/Concerns/ProvidesBrowser.php:93
/Users/miki/Projects/orchestra/testbench-dusk/vendor/laravel/dusk/src/Concerns/ProvidesBrowser.php:64
/Users/miki/Projects/orchestra/testbench-dusk/tests/Browser/AuthTest.php:30

Steps To Reproduce:
I can only replicate the issue on Mac at the moment, tests on Travis-CI running fine at the moment:
./vendor/bin/dusk-updater detect
 ----------------- -----------
  Installed         Version
 ----------------- -----------
  Chrome/Chromium   79.0.3945
  ChromeDriver      79.0.3945
 ----------------- -----------

Nevermind, no longer able to recreate this.

I am facing the same issue.

Please see https://github.com/orchestral/testbench-dusk/blob/master/README.md#chrome-versions
Which should sort this issue for you.

@Keoghan I have already run the commands but I still get the issue.

I'm running vendor/bin/dusk-updater detect --auto-update which does seem to work (with Github actions atleast). The other update command didn't work.


fix(readme.md): readme.md had bad formatting

There were the following issues with your Pull Request

Commit: dddc7b00557131be962fa310c3849a93787dafd6
Your commit message does not contain a body

Guidelines are available at https://github.com/fabric8io/fabric8-ui

This message was auto-generated by https://gitcop.com

:tada: This issue has been resolved in version 4.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Gpu particle feedback

supernova feedback code

Some checks are failing because we need to install hipRAND (I believe Orlando is already working on the other errors that arise).
I believe this command (really, a series of commands) installs hipRAND (tested using Docker):
    git clone https://github.com/ROCmSoftwarePlatform/rocRAND.git && \
    cd rocRAND; mkdir build; cd build && \
    git submodule update --init && \
    cmake -DCMAKE_CXX_COMPILER=hipcc ../. && \
    make -j2 && \
    make install

but this might be a good time to switch over to Docker containers since building hipRAND takes a long time to build and I cannot find a pre-compiled binary. I've uploaded the necessary hip docker container already: alwinm/cholla:hip_v0.1
Depending on how we want to move forward, I could push my container-based version of build_tests.yml to gpu_particle_feedback, or to dev

@alwinm If you have a docker container ready, I think this is a good time to deploy it. Why don't you go ahead and submit it directly to dev, and that way this PR will pass the build tests self-consistently.

I also agree with a couple of caveats

Give me the afternoon to figure out a more elegant way to install hipRAND, I think it comes with their apt repos but I need to check
We need a way to know which versions of everything a particular action runs with. I think this is already in you script but some simple command --version ought to do it
We need a simple way of updating the container for new versions of the various tools. A script would be perfect, even if we need to run it manually; we can discuss automating it later.

I checked. You should be able to install rocrand (which should include hiprand) the same way that rocfft is installed. Make sure to include the version you want so that you get the correct version, by default you get the latest version but we want the version that matches the other ROCm stuff.


Possibility of panics when disconnecting while messages are being constructed

evseEntity and evEntity are set to nil when the EVSE got disconnected or the EV unplugged. At the very same time the values may be used in another thread to construct a SPINE message or to gather data which would lead to a panic.

This is fixed by https://github.com/enbility/cemd/commit/23dc02dea557565014bb1f9fd69018dba6d2c7dd
The CEM does not hold any pointers to remote devices any more. Instead the event driven API always provides the remote entity the event is coming from, and subsequent actions can use that pointer.


[BUG]: Lower quality than the examples on the demo page

Describe the bug
After following the installation instructions (plus replacing phonemizer with https://github.com/justinjohn0306/phonemizer to make it work on Win 11), and using the same examples from the demo page, I was unable to replicate the quality of the examples. For example, the whispering voice always outputs something between a whisper and a normal voice. I tried both the inference script and the Gradio app, with the same result. Additionally, the duration calculator seems to be broken for Chinese - it makes the output twice as fast when set to auto.
This is the demo page result:
https://vocaroo.com/15JxVNPRScwD
This is mine:
https://vocaroo.com/13b14dZCkNau
How To Reproduce
Steps to reproduce the behavior:
Follow the instructions to install on Win 11 with special phonemizer and generate audio
Expected behavior
Quality should be the same as the examples
Screenshots

Environment Information

Operating System: Windows 11
Python Version: Python 3.10.15
Driver & CUDA Version: Driver 546.92 & CUDA 12.4
Error Messages and Logs: Posted it above, here quoted version:

./models/tts/maskgct/g2p\sources\g2p_chinese_model\poly_bert_model.onnx
Start loading: facebook/w2v-bert-2.0
D:\AIMaskGCTTTS\Amphion\models\tts\maskgct\gradio_demo.py:103: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
stat_mean_var = torch.load("./models/tts/maskgct/ckpt/wav2vec2bert_stats.pt")
D:\AIMaskGCTTTS\venv\lib\site-packages\torch\nn\utils\weight_norm.py:143: FutureWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
WeightNorm.apply(module, name, dim)
Models built successfully.
Checkpoints downloaded successfully.
Checkpoints loaded successfully.

Running on local URL:  http://<IP_ADDRESS>:7860

To create a public link, set share=True in launch().
===== New task submitted =====
Start inference...
Audio loaded.
D:\AIMaskGCTTTS\venv\lib\site-packages\whisper_init_.py:150: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
checkpoint = torch.load(fp, map_location=device)
Building prefix dict from the default dictionary ...
Loading model from cache C:\Users\Ozkan\AppData\Local\Temp\jieba.cache
Loading model cost 0.321 seconds.
Prefix dict has been built successfully.
D:\AIMaskGCTTTS\Amphion\models\tts\maskgct\g2p\g2p\chinese_model_g2p.py:100: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_new.cpp:281.)
batch_label_starts = torch.tensor(batch_label_starts, dtype=torch.long)
Saved: ./output/output_0.wav
===== New task submitted =====
Start inference...
Audio loaded.
Saved: ./output/output_1.wav

Additional context
Thank you very much for this project

I have the same problem.I followed steps on page 'https://github.com/open-mmlab/Amphion/tree/main/models/tts/maskgct', and started the gradio demo.But I found the audio generated is not as good as the official examples.I used the audio downloaded from 'https://maskgct.github.io/audios/icl_smaples/icl_10.wav' and use target text '顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。”', both in the first example of 'Zero-shot In-context Learning'.


Use fork instead of patch

Upgrade is failing due to diffs in go codegen. Is is possible we need to rebuild the go sdk?


3.8 Iteration Submenu + Notebooks

Members: Shawn Ray, Arya Savlani, Aarav Sonara

Accepted.  Mostly done well.   My biggest complaint is the name of your files, they should have unit in them to not conflict with others making lessons in you period.


Question: issues w/ jquery-ui-datepicker in component integration tests

whoops sorry about no comment before I posted:
I have a component that uses a jquery-ui-datepicker, when I run the integration tests (from ember/ember-cli 1.13) I get: TypeError: inst is undefined. The datepicker is the only component here with that issue in my specs. I have tried needs ['component:jqui-datepicker'] too to no avail. Because it's not a function is there anything special I need to be aware of?

Ember 1.13 broke the tests but the components still work.

thanks, out of curiosity which tests are you referring to?


Fails to load textures on 1.15.x servers.

The line which loads the Minecraft assets, here, doesn't correctly support all Minecraft versions.
In the Minecraft assets repo, not all versions are listed by their major version, so these fail to load. This includes 1.14.x, 1.11.x and 1.8.x
Wasn't sure if this bug should be posted in this repo, mcData, or Minecraft assets.

I noticed that too. I think in mcAsset we should use the majorVersion (that's  what we do in most packages)

Another related issue: I noticed on your screenshot https://cdn.discordapp.com/attachments/413438067471548427/731024585264463872/Screenshot_from_2020-07-09_22-49-38.png that the textures are not correctly aligned, this is due to image width being hardcoded: https://github.com/PrismarineJS/prismarine-viewer/blob/master/src/world.js#L149 It should adapt to the real image width (that depends on the version)
I only tested it for 1.13.2

Alright. Renaming the asset folders sounds like the best option for compatibility and uniformity purposes.

No I think the folders are fine (better keep the data in the true version
folder) but something (node minecraft assets or this package) should use
the version to major version mapping from mcdata
On Fri, Jul 10, 2020, 10:15 TheDudeFromCI<EMAIL_ADDRESS>wrote:

Alright. Renaming the asset folders sounds like the best option for
compatibility and uniformity purposes.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/PrismarineJS/prismarine-viewer/issues/8#issuecomment-656549027,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437WSON2EJK5N5QW5DQLR23EZXANCNFSM4OWHLFHQ
.

I think we meant changing the mapping to use major version as a key here: https://github.com/PrismarineJS/node-minecraft-assets/blob/master/index.js#L49


How can the results generated by ‘terragrunt plan’  be converted into json

Hello author, thank you for sharing your code and gaining some knowledge from your code. I would like to ask you how to convert the results generated by plan into json

import tftest env = {     "ARM_SUBSCRIPTION_ID": "x",     "ARM_CLIENT_ID": "x",     "ARM_CLIENT_SECRET": "x",     "ARM_TENANT_ID": "x",     "BLOB_ACCESS_KEY": "x" } fixtures_dir = 'salt-avaset/parameters/virtual_machines' tg = tftest.TerragruntTest(fixtures_dir, env=env, tg_run_all=True) tf_vars = {"admin_password": "x"} result = tg.plan(output=True,tg_non_interactive=True,tf_vars=tf_vars) print(tg.output())
tftest.TerraformTestError: Error decoding plan output: Expecting value: line 1 column 2 (char 1)
I have not been able to find the reason, I hope I can get your help, thank you

tftest.TerraformTestError: Error decoding plan output: Expecting value: line 1 column 2 (char 1)

import tftest
env = {
    "ARM_SUBSCRIPTION_ID": "x",
    "ARM_CLIENT_ID": "x",
    "ARM_CLIENT_SECRET": "x",
    "ARM_TENANT_ID": "x1",
    "BLOB_ACCESS_KEY": "x"
}
fixtures_dir = 'salt-avaset/parameters/virtual_machines'
tg = tftest.TerragruntTest(fixtures_dir, env=env, tg_run_all=True)
tf_vars = {"admin_password": "x"}
result = tg.plan(output=True,tg_non_interactive=True,tf_vars=tf_vars)
print(tg.output())

I have not been able to find the reason, I hope I can get your help, thank you!

hi I might get a chance to take a look but first of all https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/ - the run-all plan doesnt work all the time.  Does it work when you run it from cmd line not using tftest?
secondly, where is it failing?  in the tg.plan or tg.output

hi I might get a chance to take a look but first of all https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/ - the run-all plan doesnt work all the time. Does it work when you run it from cmd line not using tftest?
secondly, where is it failing? in the tg.plan or tg.output

I am glad to hear from you.
An exception occurred during tg.plan
Exception prompt:“raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)” and “raise TerraformTestError('Error decoding plan output: {}'.format(e))
tftest.TerraformTestError: Error decoding plan output: Expecting value: line 1 column 2 (char 1)
”

hi I might get a chance to take a look but first of all https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/ - the run-all plan doesnt work all the time. Does it work when you run it from cmd line not using tftest?
secondly, where is it failing? in the tg.plan or tg.output

I run the command(terragrunt plan) directly locally without any problems

ok thanks also what are the versions of terragrunt and terraform you are using?

ok thanks also what are the versions of terragrunt and terraform you are using?

Finally waiting for your reply, thank you。
Terraform v0.11.14 ，terragrunt version v0.18.6

@Sjenrey tftest is only compatible with Terraform >= 0.12. And thanks @davidtam for jumping on this of course!
I'll mark this as invalid, unless David wants to keep this open.

@ludoo Thank you for your reply, my English is not good, so it will be a bit difficult to read your document, I will try to upgrade the version, thank you again for your reply, thank you

@Sjenrey apologize but yeah what @ludoo said.  @ludoo yes pls close thanks!

@davidtam What do I need to do? ^_^


Ensure audit fields are populated

Add configuration so that the audit fields are populated in the domain - fields such as createdBy, lastModifiedBy, modifiedOn, createdOn.

Codecov Report

Merging #7 into master will increase coverage by 2.75%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master       #7      +/-   ##
============================================
+ Coverage     85.32%   88.07%   +2.75%
- Complexity       36       38       +2
============================================
  Files            11       11
  Lines           109      109
  Branches         13       13
============================================
+ Hits             93       96       +3
+ Misses           13        8       -5
- Partials          3        5       +2

Impacted Files
Coverage Δ
Complexity Δ

...notation/service/security/SecurityUtilService.java
56.25% <0%> (+18.75%)
5% <0%> (+2%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62cd98c...353279e. Read the comment docs.


devbox: 0.3.2 -> 0.3.3

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/jetpack-io/devbox/releases.
meta.description for devbox is: Instant, easy, predictable shells and containers.
meta.homepage for devbox is: https://www.jetpack.io/devbox
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.3.3 with grep in /nix/store/fvv84c5rag77p3sybyga0rjjvrvn0k3h-devbox-0.3.3
found 0.3.3 in filename of file in /nix/store/fvv84c5rag77p3sybyga0rjjvrvn0k3h-devbox-0.3.3

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
devbox

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/fvv84c5rag77p3sybyga0rjjvrvn0k3h-devbox-0.3.3 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A devbox https://github.com/r-ryantm/nixpkgs/archive/31f57de7dd85b7575a01e20b0be2ae7b7fbd70df.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/fvv84c5rag77p3sybyga0rjjvrvn0k3h-devbox-0.3.3
ls -la /nix/store/fvv84c5rag77p3sybyga0rjjvrvn0k3h-devbox-0.3.3/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

devbox

Maintainer pings
cc @urandom2 for testing.

Result of nixpkgs-review pr 215051 run on aarch64-darwin 1

1 package built:

devbox


Returning array of values or of objects

Using
db: new alasql.Database('tableName')
I experience that db.exec sometimes gives me an array of values and sometimes an array of objects with values.
First I understood it as returning an array of values if only one column was selected. But the same query with a DISTINCT returns an array of objects.

I tried to replicate the issue but did not succed ( http://jsfiddle.net/rndy99jy/ )
Do you have clues of why it would ever return an array of values?

Any news on this one?

It is strange...
By specification, it has to provide always array of objects, except if you are using modificators like:
    SELECT VALUE mycol FROM mytable -- returns a value of first column of first line
    SELECT COLUMN mycol FROM mytable -- returns array of values from first column
    SELECT ROW mycol FROM mytable -- returns array of values from first row
    SELECT MATRIX mycol FROM mytable -- returns array of arrays

If this is not an answer, we need to develop example to find the source of the problem.

Now you example works fine. http://jsfiddle.net/rndy99jy/ Probably the problem is fixed :)

Yayyy :)


Chapter 4: cannot assign 'torch.nn.modules.container.Sequential' as child module

I just downgraded my torch and other packages to the correct version. Seems this doesn't work.

Could you please elaborate a bit more on this problem? It might help if you show us the stacktrace and the versions you used. If you use torch < 1.8.x it might be due to wrong depencies to torchvision etc.. Have a look here for the correct versions to use
I just ran chapter4.ipynb with the current versions torch 1.8.1 (+cu101) and torchvision 0.9.1 (+cu101) - and it's running without any problems. The same when I use torch==1.6.0 and torchvision==0.7.0.

At first, I tried it with torch 1.8.1(+cpu) and torchvision 0.9.1(+cpu), I got this error:
I thought it might due to the torch version, then I tried downgrade torch&torchvision, however I still have the same problem.
I googled but found nothing helpful.
I don't think it's the problem of cpu version torch.
help,plz.
------------------ 原始邮件 ------------------
发件人:                                                                                "falloutdurham/beginners-pytorch-deep-learning"                                                                                @.>;
发送时间: 2021年6月4日(星期五) 晚上8:54
@.>;
<EMAIL_ADDRESS>主题: Re: [falloutdurham/beginners-pytorch-deep-learning] Chapter 4: cannot assign 'torch.nn.modules.container.Sequential' as child module (#64)
Could you please elaborate a bit more on this problem? It might help if you show us the stacktrace and the versions you used. If you use torch < 1.8.x it might be due to wrong depencies to torchvision etc.. Have a look here for the correct versions to use
I just ran chapter4.ipynb with the current versions torch 1.8.1 (+cu101) and torchvision 0.9.1 (+cu101) - and it's running without any problems. The same when I use torch==1.6.0 and torchvision==0.7.0.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Do you run the code on your local machine? Have you tried it on Colab?
At which line does the code break?
transfer_model.fc = nn.Sequential(nn.Linear(transfer_model.fc.in_features,500),
                                  nn.ReLU(), nn.Dropout(), nn.Linear(500,2))

Here? Have you deleted a character (comma, bracket)  by incidence?
Is cannot assign 'torch.nn.modules.container.Sequential' as child module the only message you receive? That's a TypeError, but yet I cannot figure out where it stems from as I don't revceive the message.
Has the pretrained model been loaded correctly? (any error after excecution of transfer_model = models.resnet50(pretrained=True)?)

I downloaded the original version of your tutorial from github, and I didn't get a syntax error. The pretrained model was loaded correctly. So I assume there is nothing wrong with the code.
I tried to reconstruct other layers of Resnet50 like this:
Then I get the same TypeError:
And then I tried to replace the fc layer with a single ReLU function:
It worked!
A single function to reconstruct the layer, OK!A sequential to replace the layer, NOT OK!
I assume this is some kinda unknown bug from pytorch, or it's just my machine's problem. lol
------------------ 原始邮件 ------------------
发件人: @.>;
发送时间: 2021年6月5日(星期六) 下午3:29
收件人: @.>;
抄送: @.>; @.>;
主题: Re: [falloutdurham/beginners-pytorch-deep-learning] Chapter 4: cannot assign 'torch.nn.modules.container.Sequential' as child module (#64)
Do you run the code on your local machine? Have you tried it on Colab?
At which line does the code break?
transfer_model.fc = nn.Sequential(nn.Linear(transfer_model.fc.in_features,500),                                   nn.ReLU(), nn.Dropout(), nn.Linear(500,2))
Here? Have you deleted a character (comma, bracket)  by incidence?
Is cannot assign 'torch.nn.modules.container.Sequential' as child module the only message you receive? That's a TypeError, but yet I cannot figure out where it stems from as I don't revceive the message.
Has the pretrained model been loaded correctly? (any error after excecution of transfer_model = models.resnet50(pretrained=True)?)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

never mind, it worked somehow.
------------------ 原始邮件 ------------------
发件人:                                                                                "falloutdurham/beginners-pytorch-deep-learning"                                                                                @.>;
发送时间: 2021年6月5日(星期六) 下午3:29
@.>;
<EMAIL_ADDRESS>主题: Re: [falloutdurham/beginners-pytorch-deep-learning] Chapter 4: cannot assign 'torch.nn.modules.container.Sequential' as child module (#64)
Do you run the code on your local machine? Have you tried it on Colab?
At which line does the code break?
transfer_model.fc = nn.Sequential(nn.Linear(transfer_model.fc.in_features,500),                                   nn.ReLU(), nn.Dropout(), nn.Linear(500,2))
Here? Have you deleted a character (comma, bracket)  by incidence?
Is cannot assign 'torch.nn.modules.container.Sequential' as child module the only message you receive? That's a TypeError, but yet I cannot figure out where it stems from as I don't revceive the message.
Has the pretrained model been loaded correctly? (any error after excecution of transfer_model = models.resnet50(pretrained=True)?)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

OK.
I think I get what you mean, but unfortunately it seems that in your comment some explanations (after the respective colons) are missing/not printed.


Default admin password

Hi,
Once the plugin is installed and the  webadmin user ID is entered what is the correct password?
I am using version 1.0.17 of the plugin.
Caddy is v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=
I am starting caddy with user caddy:caddy by use of systemd and writing the initial users.json to /var/lib/caddy successfully. Thus;
{
"version": "1.0.17",
"policy": {
"password": {
"keep_versions": 10,
"min_length": 8,
"max_length": 128,
"require_uppercase": false,
"require_lowercase": false,
"require_number": false,
"require_non_alpha_numeric": false,
"block_reuse": false,
"block_password_change": false
},
"user": {
"min_length": 3,
"max_length": 50,
"allow_non_alpha_numeric": false,
"allow_uppercase": false
}
},
"revision": 2,
"last_modified": "2022-02-28T17:05:47.02932396Z",
"users": [
{
"id": "2e6c223e-ef2b-4036-bcc6-8f562ce21692",
"username": "webadmin",
"email_address": {
"address"<EMAIL_ADDRESS>"domain": "localdomain.local"
},
"email_addresses": [
{
"address"<EMAIL_ADDRESS>"domain": "localdomain.local"
}
],
"passwords": [
{
"purpose": "generic",
"algorithm": "bcrypt",
"hash": "$2a$10$VvjXt5P.tEh25zJpvlL/fePoJ5aChHk389eDXqkMZaDz/1p6DyPL2",
"cost": 10,
"expired_at": "0001-01-01T00:00:00Z",
"created_at": "2022-02-28T17:05:46.942999899Z",
"disabled_at": "0001-01-01T00:00:00Z"
}
],
"created": "2022-02-28T17:05:46.942998945Z",
"last_modified": "2022-02-28T17:05:47.0293181Z",
"roles": [
{
"name": "admin",
"organization": "authp"
}
]
}
]
}
without me being able to initially login as webadmin to make config cahnges I think I am stuck.
Thanks in advance for some pointers.
brian

@brinycoolie , remove users.json and restart the server. After the restart, you will see the log entry that would have<EMAIL_ADDRESS>and the password for the webadmin user. It is a random password. After that, login and change the password.
Alternatively, see https://authp.github.io/docs/authenticate/local/password-management to change the hash you have, i.e. $2a$10$VvjXt5P.tEh25zJpvlL/fePoJ5aChHk389eDXqkMZaDz/1p6DyPL2 to something that you would generate with bcrypt-cli.

Thanks Paul,
I did not see that the hash was randomly generated each creation, so I assumed it was hardcoded.
Also I had quite a hard time getting my tired old head around the new paradigm with Caddy log files so I could even see the  new random hash. My bad! Got there in the end.
I wanted to avoid, if possible deleting the entire file since it would also delete any users there so I wanted to work on the second option. I found that the go bitnami implementation would not install on my system.
see:
go version go1.13.8 linux/amd64
brian@beetle-1:~$ go get -u github.com/bitnami/bcrypt-cli
then, bcrypt-cli --help
bcrypt-cli: command not found
brian@beetle-1:~$ bcrypt
Command 'bcrypt' not found, did you mean:
command 'ccrypt' from deb ccrypt (1.11-1)
command 'crypt' from deb mcrypt (2.6.8-4)
command 'zcrypt' from deb barnowl (1.10-1build1)
command 'mcrypt' from deb mcrypt (2.6.8-4)
command 'scrypt' from deb scrypt (1.3.0-3)
Try: sudo apt install
So I experimented with several javascript based bcrypt command line tools and they did not crosscheck with the go version implemented in the security plugin, so I continued to experiment and tried the htpasswd from apache-utils and it seems to crosscheck and roundtrip well with the security plugin. Even to (at least) two cost functions 10 and 16. however 18 did not finish in a timely manner.
Now I have a bit of flexibility with the users file.
Thanks again.
brian


How do I build this project.

As the title of the issue says, how do I build this project? I am trying to make it as a gradle project, however, the examples wouldn't compile. My gradle file looks like this -
apply plugin: 'java'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
    jcenter()
    maven { url "https://jitpack.io" }
}

dependencies {
    compile 'org.dyn4j:dyn4j:3.2.3'
    compile 'org.jogamp.jogl:jogl-all:2.3.1'
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

I always get this error while building the project -
Error:(174, 21) java: cannot access com.jogamp.common.type.WriteCloneable
  class file for com.jogamp.common.type.WriteCloneable not found

I'm not familiar enough with grade to offer any advise there.  I build in eclipse.  I don't remember what version of JOGL I last built it with, but that may be the problem.  I've had to make minor changes whenever I update JOGL.

Could you please mention the jogl version in the README file, I was trying to port it to gradle so that it would be easier to build in Intellij Idea

Ok, looking at this closer, I think you need to include the gluegen jars:

gluegen.jar
gluegen-rt.jar

You'll need the natives of gluegen and jogl if you want to run it.  2.3.1 is the version that I'm building against.  I will add to the Readme.

Hey that works! My build.gradle for the examples module is like this -
version '3.2.3'

apply plugin: 'java'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
    jcenter()
    maven { url "https://jitpack.io" }
}

dependencies {
    compile 'org.dyn4j:dyn4j:3.2.3'
    compile 'org.jogamp.gluegen:gluegen:2.3.1'
    compile 'org.jogamp.gluegen:gluegen-rt:2.3.1'
    compile 'org.jogamp.jogl:jogl-all:2.3.1'
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

Thanks!!


Add fuzzy tests to noir

I think it would be interesting to incorporate fuzz testing for the type checker. I came across a resource that might be helpful in this regard: https://hardekbc.github.io/files/dewey15fuzzing.pdf

Getting them fixed as part of this issue: https://github.com/noir-lang/noir/issues/2881


fix Unable to delete a field in a module

Describe the bug
When deleting a field in a module, it first disappears but immediately comes back after saving the module.
To Reproduce
Steps to reproduce the behavior:

Go to a module
Click on the trash next to a field and confirm the deletion
Save the module
The field is back again.

Expected behavior
The field is deleted.
[Screenshots]

Desktop (please complete the following information):

OS: Windows 10
Browser Chrome
Version 88.0.4324.190

I've addressed this bug a little earlier today. It will be released in the next patch release early next week.


refactor: theme setup

Based on https://github.com/curvefi/curve-frontend/pull/421
Do not read the JSON file during runtime. Instead, generate a TypeScript file.
Get rid of FSD structure for themes
Get rid of import forwarding to MUI, any customization should be done via theme

Storybook is failing because we need to enable github pages. That shouldn't block the merge


luci-theme-material: fix wrong active state on common prefix node

Before fixed, if we have two nodes: 'services/ddns' and 'services/ddnsto',
click any one of they, will show they all actived.
Signed-off-by: Liangbin Lian<EMAIL_ADDRESS>(cherry picked from commit 97d50d2c6b80805cd7e513eeafc8b62fef4ab1b6)

17.01 is deprecated and unsupported. EOL


DelegatedHaxballRoom: added dynamic method integration

Added dynamic method integration and removed static one.

You should also make changes in haxilium.js file because I use super keyword there. You have to replace super with this._room.

I don't think I need to replace super, because i assigned those methods to the class itself.

You can checkout to that commit and see if it works.

Thanks, merged. Your way of delegating methods is very cool.


[build] Put decoders in single decode library

@Alexandr-Konovalov,
Could u please provide a justification why do we need to proceed this way? For me, another approach -  ([build] Separate component build to dedicated CMakeLists) is much better.

@Alexandr-Konovalov,
Could u please provide a justification why do we need to proceed this way? For me, another approach - ([build] Separate component build to dedicated CMakeLists) is much better.

@dmitry-gurulev , I think that approach "one component - one CMakeLists.txt" is more "structured", so to say, and dislike multiple unrelated include_directories. But @dvrogozh is sure that having single big library is simplifying modifications, if I understand him correctly.

@dmitry-gurulev : yeah, that's trade off. Few bigger cmake files are times easier to support as it turns out.

Could you please rebase to catch up infra changes?


iOS native test runner should provide way of specifying tests to run on CI

We need some JSON file (the same as XCTest framework version for swift) that runner will use to identify the tests to be run.
@yaronyg please update the issue priority.
Adding @mlesnic into notify list as a person who needs it.

We can just replace contents with empty contents of files with test that we don't need. @mlesnic could you please try it.

Just replacing the file contents will take 3 days? But one way or another we HAVE to do this. We have way too many problems with CI and we need a known good solid test we can run and be sure it will tell us if CI is happy or not. Otherwise everyone is just going to blame CI for everything.

'3 days' were before we decided to replace contents of files. This estimation was for list of tests in JSON.


[Core] Add interpretability capabilities trough SHAP

Description
To enhance the interpretability of models trained using MLForecast, we propose leveraging SHAP (SHapley Additive exPlanations). SHAP is compatible with XGBoost, LightGBM, and scikit-learn models. Currently, if we want to use it, we need to create the dataset for which we desire forecast explanations (using preprocess) and iterate over each trained model using the following:
explainer = shap.Explainer(model)
shap_values = explainer(X)

The goal is to introduce a method, possibly named shap_values, to generate SHAP values for the forecasts from all trained models.
Use case
No response

Currently, if we want to use it, we need to create the dataset for which we desire forecast explanations (using preprocess) and iterate over each trained model

I am not sure if this feature will be released soon, but in the meantime I cannot seem to use SHAP values using the above method.
model = MLForecast(models=[xgb.XGBRegressor()], freq=freq)
model.fit(df)
explainer = shap.Explainer(list(model.models.values())[0])

    169             algorithm = "permutation"
    171     # if we get here then we don't know how to handle what was given to us
    172     else:
--> 173         raise TypeError("The passed model is not callable and cannot be analyzed directly with the given masker! Model: " + str(model))
    175 # build the right subclass
    176 if algorithm == "exact":

TypeError: The passed model is not callable and cannot be analyzed directly with the given masker! Model: XGBRegressor(base_score=None, booster=None, callbacks=None,
             colsample_bylevel=None, colsample_bynode=None,
             colsample_bytree=None, early_stopping_rounds=None,
             enable_categorical=False, eval_metric=None, feature_types=None,
             gamma=None, gpu_id=None, grow_policy=None, importance_type=None,
             interaction_constraints=None, learning_rate=None, max_bin=None,
             max_cat_threshold=None, max_cat_to_onehot=None,
             max_delta_step=None, max_depth=None, max_leaves=None,
             min_child_weight=None, missing=nan, monotone_constraints=None,
             n_estimators=100, n_jobs=None, num_parallel_tree=None,
             predictor=None, random_state=None, ...)

Perhaps it has something to do with how the models I retrieve using model.models are not fitted, even after running model.fit(df):
list(model.models.values())[0].predict(df)
    646 if not self.__sklearn_is_fitted__():
    647     from sklearn.exceptions import NotFittedError
--> 649     raise NotFittedError("need to call fit or load_model beforehand")
    650 return self._Booster

NotFittedError: need to call fit or load_model beforehand

Hey @elisevansartefact. The fitted models are stored in the models_ attribute, so something like shap.Explainer(model.models_['XGBRegressor']) should work. The models argument accepts a dict as well (in case you prefer a different name for the model).
Also if you're looking to get explanations on the predictions you'll need to store the features used in time step, you may find this thread useful. Here's the relevant snippet:
from functools import partial

def extract_features(df, save_list):
 save_list.append(df)
 return df

save_list = []
extract_features_callback = partial(extract_features, save_list=save_list)
fcst.predict(..., before_predict_callback=extract_features_callback)
features = pd.concat(save_list)

Would probably also want to use https://github.com/linkedin/fasttreeshap instead

Hey folks. We've added a guide which explains how to get the trained models and compute the SHAP values for training and inference. I think this gives full control on how to compute them (sample size, etc). Please let us know if you'd prefer something integrated into the library.

@jmoralez the guide makes this a lot easier but its probably worth noting that it only works for single model recursive fits. If a model (or models) is fitted with a direct strategy then each model in the list has a different explainer.


Pinmappings do not reflect Serial Tx/Rx available from 10586 on

https://ms-iot.github.io/content/en-US/win10/samples/PinMappingsRPi2.htm
Pins 8 & 10 are still marked 'Reserved'.

Thanks! Pin mappings have been updated to reflect the addition of Serial UART.
https://ms-iot.github.io/content/en-US/win10/samples/PinMappingsRPi2.htm


Support ET Serie 29,9kW with 2 Batteries

Hello, is it possible to include the new GoodWe WR ET 29.9. Here the 2nd battery connection would have to be added. Furthermore, the inverter has 6 PV connections with 3 MPPT.
Thank you in advance

Oooo, that looks like some larger beast, no longer small residential inverter.
In theory yes, it might be possible to extend support for that, but there are few things to verify/test and I would need your cooperation there.
Open a related ticket in the underlying library - https://github.com/marcelblijleven/goodwe and it would be helpful if you'll setup python environment so you can execute some sample commands from there - the inverter_check.py

Many thanks for the quick response.
I can't really program, only copy & paste works... :-) I'll get an environment installed and I'll probably be able to run scripts, but then that's it....
Von: mletenay @.>
Gesendet: Montag, 13. März 2023 18:01
An: mletenay/home-assistant-goodwe-inverter @.>
Cc: Christian Schmitz @.>; Author @.>
Betreff: Re: [mletenay/home-assistant-goodwe-inverter] Support ET Serie 29,9kW with 2 Batteries (Issue #172)
Oooo, that looks like some larger beast, no longer small residential inverter.
In theory yes, it might be possible to extend support for that, but there are few things to verify/test and I would need your cooperation there.
Open a related ticket in the underlying library - https://github.com/marcelblijleven/goodwe and it would be helpful if you'll setup python environment so you can execute some sample commands from there - the inverter_check.py
—
Reply to this email directly, view it on GitHubhttps://github.com/mletenay/home-assistant-goodwe-inverter/issues/172#issuecomment-1466535025, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2OEOUWY4ETUPFBSCFDFVELW35HFZANCNFSM6AAAAAAVZIQC7U.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

That should be good enough, just install python and checkout the library repo.
I'll give you instructions how to proceed, in best case we'll need just few tweaks.

I have some errors in the inverter software.
E.g. the 2nd battery is not displayed correctly. I'm waiting for a new firmware to be installed. As soon as this is done, I'll let you know
Thank you in advance

ET >15 incl. 2nd battery is now properly supported.


[Dataset | Model card] When pushing to template repos, work on actual raw contents

Question: do we actually want this feature or not?
Internal Slack convo
cc @Wauplin
Details
Generated commit for model cards (would need to do the same for datasets): https://huggingface.co/templates/model-card-example/commit/901deccf5acead553f9b082aca480d966e61f355

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@julien-c  I get the idea of the PR. Not sure what is the optimal way to add a comment to the modelcard. I'm fine with this solution if we don't have any better simple solution. My only concern is that if we have modelcards with a comment in the metadata section, the modelcards module is still unable to process them. So if a user uses the ModelCard feature of hfh at some point, it will remove the comment (is it really an issue though ?).
In any case, if we go with this solution, I would add a comment in push_repocard_example to explain shortly why we are not using the module.

About the failing tests, I don't know what the issue is. I encountered it as well on Friday 23rd afternoon but weren't sure if it was my fault or not. It seems that on staging we cannot commit a README.md file from the HTTP endpoint. It gets a HTTP 400 Bad request for commit endpoint: fetch failed from moon-landing. It's not about the content of the file but really the filename (I tried it manually and any other filename can be uploaded without any problem.). Maybe something to restart on staging ? cc @julien-c @Pierrci

maybe ping @SBrandeis about the staging env failure (no rush though)

Sleeping on this PR a bit, I think it's an okay approach. No big deal if the actual ModelCard strips comments in metadata (which is expected, I guess – YAML => objects conversion doesn't have a concept of comments)
BTW we need to do it for dataset cards too.
Pinging @nateraw for his opinion though, just to be sure

@nateraw @julien-c I have added both model_summary and dataset_summary to explain from where the templates come from. Could you quickly review and we merge it ? :)


how to- convert timestamp to date

hi brains
is there any way to convert timestamp to date?
I wanna create date filter, but have timestamp field
10X

What backend are you using? You can use a calculated column to cast your timestamp to a date. (Edit Table -> List Table Column -> (+) -> Put expression in "Expression"

great
do you know why I cant filter by this custom field?
I get the following error:

10x

Notice: this issue has been closed because it has been inactive for 621 days. Feel free to comment and request for this issue to be reopened.