[Request] Support Android client

Hi there,
I am relatively new to SoftEther but it is fast and simple to set up and offers a lot of great features.
I have everything up and running but the one issue I have is the lack of any proper Android support. More and more SmartTVs use Android, FireTV uses Android and basically all good phones use Android ;-)
I was hoping that there would be support for Android devices as the built in VPN client is pretty useless and won't work properly (especially if not using port 443).
Maybe there could be an Android version that can also be sideloaded and used on FireTVs and SmartTVs?
P.S.: Yes, I have posted on the forum but cannot find any real solutions anywhere. And I am pretty sure a decent client software like the Windows one would aolve all issues quick and easy :-)

hello, @AleXSR700 , please define next step

Well, a deprecated SoftEther Android version is available on the play store. However, it requires root.
I contacted the developer who has given up on the project. I assume this is because it requires root.
We would need someone with sufficient knowledge to implement the necessary adjustments to be able to develop a non-root version. Once this is the case I think more people will start compiling.
But right now SoftEther is basically unusable with Android. OpenVPN does not work properly either. And I followed the tutorial but still it does not work.
We need an Android version of the SoftEther client.

Hi,
What kind of issues are you encountering with OpenVPN?
About root permissions: they are required because Android API's doesn't support TAP devices.
We can workaround the limitation by emulating a TAP device, as guizmovpn does.
We actually already have a TUN -> TAP conversion layer, but it's server-side.

Hi there,
I am unable to get my Android phone (Samsung S10 Plus) to connect to my VPN server. I have tried everything I can think of and anything that was recommended or tutored anywhere.
My port 443 is closed as it is being used by something I cannot configure. My server is a NAS that sits behind a cable internet router (DS Lite) which cannot be configured to do port translation.
But since my Windows notebook works without any issues, it obviously is a Android/lack of software issue.
I tried both OpenVPN connect as well as OpenVPN for Android. The latter seems much better as it allows you to play with the settings after importing the opnv file.
I opened a thread if you are interested in logs.
https://www.vpnusers.com/viewtopic.php?f=7&t=64719&sid=9b1fb80256c5f85c15d98e06014caf24
I am open to any help or ideas, but it feels like a proper dedicated app that actually makes use of everything SoftEther has to offer would be the best long run solution. Especially since SoftEther itself claims that the other protocols/VPNs like OpenVPN are inferior.

In fact, I have tested the google store program, it is very difficult to use, not every connection will be successful, you must always try, mobile phone ROOT can not be used later, in fact, I always want to have similar tools for people familiar with Android

If you want to use a mobile phone, you may need to change to other solutions, such as routers. I already have successful cases. This is very complicated and requires a lot of cost

Hi,
What kind of issues are you encountering with OpenVPN?
About root permissions: they are required because Android's API doesn't support TAP devices.
We can workaround the limitation by emulating a TAP device, as guizmovpn does.
We actually already have a TUN -> TAP conversion layer, but it's server-side.

Can you point out the specific location of this TUN -> TAP code in guizmovpn?
https://github.com/SoftEtherVPN/SoftEtherVPN/blob/44447e9762243c7d3818b0440bc4b5547f6b08b9/src/Cedar/Connection.c#L2005
Is there any documentation for the specific protocol for converting the tcp stream to BLOCK?
thank you very much

Hi,
What kind of issues are you encountering with OpenVPN?
About root permissions: they are required because Android's API doesn't support TAP devices.
We can workaround the limitation by emulating a TAP device, as guizmovpn does.
We actually already have a TUN -> TAP conversion layer, but it's server-side.

Can you point out the specific location of this TUN -> TAP code in guizmovpn?
https://github.com/SoftEtherVPN/SoftEtherVPN/blob/44447e9762243c7d3818b0440bc4b5547f6b08b9/src/Cedar/Connection.c#L2005
Is there any documentation for the specific protocol for converting the tcp stream to BLOCK?
thank you very much

Can you point out the specific location of this TUN -> TAP code in guizmovpn?

https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu.c
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu.h
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu_dhcp.c
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu_dhcp.h

Is there any documentation for the specific protocol for converting the tcp stream to BLOCK?

No documentation, but I can help you. What are you attempting to do?

Can you point out the specific location of this TUN -> TAP code in guizmovpn?

https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu.c
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu.h
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu_dhcp.c
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu_dhcp.h

Is there any documentation for the specific protocol for converting the tcp stream to BLOCK?

No documentation, but I can help you. What are you attempting to do?

Can you point out the specific location of this TUN -> TAP code in guizmovpn?

https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu.c
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu.h
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu_dhcp.c
https://github.com/SoftEtherVPN/guizmovpn/blob/master/openvpn/tapemu_dhcp.h

Is there any documentation for the specific protocol for converting the tcp stream to BLOCK?

No documentation, but I can help you. What are you attempting to do?
Then I will look at the code myself. Among them, the ssl-vpn protocol means that the tap layer (data link layer) packets are forwarded between the client and the server through the tcp data stream?
Thank you

Correct.

Correct.
https://github.com/SoftEtherVPN/guizmovpn/blob/47e36ab72fbb311e999cf38eb4e5ec03774f2dc2/openvpn/tapemu.c#L158
The pcap_inject function is used here. This function requires pcap. Does the Android system pcap also require root privileges？
Thank you

Hey, I can develop the android version of Ethernet over HTTPS protocol client.
can you assign this issue to me ?
@davidebeatrici

I already tried to do it but some of the packets received from the server (with no udp acceleration and no SSL Encryption for ethernet frames) doesn't have this packet structure:
4 bytes containing Packet count, 4 bytes size, n bytes packet ...
what are these ? I also tried to read code but couldn't find anything about this

I have 2 questions:

I have attempted to implement SE protocol where I receive packets from a server that are structured without UDP acceleration and SSL encryption for Ethernet frames. However, some of the packets do not adhere to the expected structure of 4 bytes for the packet count, followed by 4 bytes indicating the size, and then the Ethernet frame (n bytes). I could not find any reference to this in the code I reviewed. What could these discrepancies indicate? (e.g. I captured a packet in wireshark and it was started with these bytes: 2E4A112334DE2D63)

From my understanding, the Cedar software receives data from sockets into buffers of size 65535 bytes. What happens if the size of an Ethernet frame in the incoming packets exceeds this buffer size?

@chipitsine @davidebeatrici

I'm only confused of these things.

@b00tkitism , I forgot to mention android client fork https://github.com/kittoku/Minimum-VPN-Client-for-SoftEther-VPN

@chipitsine,
Thank you for your response. I have already disabled SSL encryption for tunneling, and while the packets are generally sent and received without encryption, I occasionally encounter some packets that I cannot identify.
Could you please provide a reference in the SoftEtherVPN source code that could help me understand what these packets are?
Additionally, I have reviewed that Android client but found it to have performance issues and a lack of customization options compared to the Windows client. This is why I am working on developing a custom client.
I appreciate your assistance and look forward to your guidance.

https://github.com/kittoku/Open-SSTP-Client
That client works pretty well with the SSTP protocol of SoftEther. It was actually written in mind of the SoftEther implementation AFAIK

@Evengard @davidebeatrici @chipitsine
Thanks for you replies, I already developed the first version of Android client with the help of Android NDK and GoMobile (It's just for the test, I'll use C after all the tests passed)
I think I have a logic error in my code (TCP Only, no udp acceleration) I'll try to write a prototype of that here:
THREAD_1:

Send keep alive packets

THREAD_2:

Continuously read packets from the tap adapter (tun in android + emulating tap with appending Ethernet Frame headers at the first of ip header)
Create a buffer and put 1 (uint32, big endian, 4 bytes) at the very first of buffer, after that append another 4 bytes indicating ethernet frame length and after that append the Ethernet Frame itself.

THREAD_3:

Continuously read from socket and append the data in a FIFO Queue.
Read the data from FIFO in a while loop using the same condition as Connection.C
After getting access to the Data Block (Ethernet Frame) write it to tap adapter (in Android, remove the Ethernet Headers and write to tun)

whats the problem here? my code works but most of the packets are lost, ping to the gateway is around 500 - 2000ms

@b00tkitism , do you build SE VPN from source or you implemented protocol by yourself from scratch ?

@chipitsine, as I said earlier, I'm trying to write a light weight client for android without performance issues. So I should write it from scratch because:

Current SE code doesn't support Android
It needs to be light weight

I'm still working on it and I don't figured it out yet, any thoughts?

@b00tkitism Please share the code with us, that way we can check whether there is a problem in it.

@davidebeatrici, can you give me a logic or check the logic that i implemented and described ? Cause my code looks like a piece of shi- and only I can understand what shi- i wrote.


tbb accelerate，python interface,undefined symbol: _ZTIN3tbb4taskE

hi, i am using dlib to run my face research. And i am try to accelerate the face detection by using TBB。As the most code is written in the cpp head files ,so it is ok to compile the c++ projects using dlib with adding TBB includes and  links. ****
However ,when i want to compile the python interface, it goes wrong. I am not clear of how to add the TBB settings to the cmake files. I add two lines to the tools/python/CMakesLists.txt before the set(python_srcs like this:
include_directories(/home/jess/soft/tbb-2018_U1/include)
link_directories(/home/jess/soft/tbb-2018_U1/build/linux_intel64_gcc_cc4.8_libc2.19_kernel4.2.0_release)
the "python setup.py install" goes OK
but i got the "linux-x86_64.egg/dlib/dlib.so: undefined symbol: _ZTIN3tbb4taskE" error while import dlib in python.

You need to add TBB to the target_link_libraries.  So you would need to edit the cmake code in dlib/cmake_utils/add_python_module since that's where target_link_libraries is called.

@davisking thanks a lot! it works. for the development of dlib, i have 2 suggestions,1 add tbb or openmp to parallel the pyramid work; 2 make the pyramid levels settable for searching if we have fixed sizes of faces from the same camera.


After migration, some icons don't load properly (sometimes)

Icon 1 doesn't appear when logged out. (http://mena.iogt.site/media/svg-to-png-maps/svg-to-png_KjEfHoY.png)
Icon 2 doesn't appear.

complete


Gringo culture

I agree it needs a better name.
However, coding it as americano would cause such culture to overlap with the same-coded one that comes from Italians.

Can't we change the Italian one?  We'll get a lot more Spanish than Italian colonisers,

Gonna do it in the 1.25 update.

Solved in #461


Should we get rid of Hydra?

Motivation
This is motivated by recent development of LightningCLI which IMO is simpler, and better supported primarily because it exists within Lightning.
Over time I've noticed that due to the changes upstream in Lightning, hydra default configs are going out of sync every so often.
I also notice that there are fundamental issues such as this https://github.com/PyTorchLightning/lightning-transformers/issues/149 where the confs exist within the package. We assume that the user will always clone the repo and make their modifications there. I would be curious if this works for users currently.
Pitch
Remove Hydra, treating this library more as a set of components people can import into their own scripts. If we'd like to keep the ability to run training and testing from the library, we should use the LightningCLI instead of Hydra.
Alternatives
Keep as is :)
cc @mathemusician @Borda @carmocca

Playing devil's advocate:

Over time I've noticed that due to the changes upstream in Lightning, hydra default configs are going out of sync every so often.

The CLI suffers from exactly this issue too. There's 2 proposals in jsonargparse to improve this a bit but they haven't been worked on at the moment:
https://github.com/omni-us/jsonargparse/issues/79
https://github.com/omni-us/jsonargparse/issues/91

I also notice that there are fundamental issues such as this #149 where the confs exist within the package. We assume that the user will always clone the repo and make their modifications there.

The CLI would also have this problem if we wanted to distribute its config files. But I guess we wouldn't need to as much if we made the switch.

treating this library more as a set of components people can import into their own scripts.

This is good regardless of the parsing library used. The choice of Hydra or the LightningCLI will come down to what the users are doing and how.

@SeanNaren @Borda Could you give a quick example of how people would use this library without hydra? If we can come up with a list of use cases where this would make the life of future users easier/simpler, then it would be a lot easier moving this issue forward.
When I started learning about this library, Hydra was the hardest thing to learn since it had the least documentation. Getting rid of it would theoretically increase its usability, because it would be one less thing to learn. I had initially thought Hydra would make things easier, but it was only easier after learning about the entire library (which took me a month).
I propose an experiment, albeit, an expensive one. We replace Hydra with LightningCLI, and if more people use it, we'll stick with it. If not, we can always go back :0 Converting this library into a set of components may be challenging but worthwhile. It might even be merged into Bolts.

Thank you both for your comments!
Addressing @carmocca comments:

The CLI suffers from exactly this issue too. There's 2 proposals in jsonargparse to improve this a bit but they haven't been worked on at the moment:

I'm unsure how these are related, it seems the second one is about trying to reproduce a saved config that was run on an earlier version of Lightning? I think this arguably could be expected behaviour. For Lightning-transformers specifically, I meant the fact that we rely on hardcoded parameters within the Trainer config. This has been outdated a few times as Lightning deprecates parameters/introduces parameters meaning users couldn't run the code out of the box unless they changed the configs manually.

This is good regardless of the parsing library used. The choice of Hydra or the LightningCLI will come down to what the users are doing and how.

Agreed, and currently this is supported (props to @carmocca who ensured that this remained). It can be seen from the test here.
The objective of this issue is to determine if we believe that making this experience a first class citizen would be beneficial for users and reduce complication by allowing users to pick what they'd like, defaulting to LightningCLI as it's the simpler of the two.
Addressing @mathemusician comments:

@SeanNaren @Borda Could you give a quick example of how people would use this library without hydra? If we can come up with a list of use cases where this would make the life of future users easier/simpler, then it would be a lot easier moving this issue forward.

The test here gives us a good sense, however still contains some ugliness due to Hydra. I imagine:
tokenizer = AutoTokenizer.from_pretrained(backbone="patrickvonplaten/t5-tiny-random")
model = TranslationTransformer(
    backbone="patrickvonplaten/t5-tiny-random"
)
dm = TranslationDataModule(
    batch_size=1,
    dataset_name="wmt16",
    dataset_config_name="ro-en",
    source_language="en",
    target_language="ro",
)
trainer = pl.Trainer()

trainer.fit(model, dm)

In this sense, we offer just LightningModules and LightningDataModules for the individuals tasks. the trainer can then be swapped out for the LightningCLI. Making this change also removes some of the boilerplate around instantiation etc. If the user wants to swap the optimizer/scheduler, they can do so via overriding configure_optimizers.
Personally I'm unsure which direction is better. I believe that simplicity of a library is important, and building frameworks rarely provide long term sustainability unless supported well (and the support for Lightning Transformers has been wishwashy, mostly my fault).
I like your idea of trying it out, seeing how the code looks like and then making a consensus whether to swap. But right now I think it would be good to ask a few more people to pitch in!

I don't want to sidetrack this conversation but to clarify:

I meant the fact that we rely on hardcoded parameters within the Trainer config. This has been outdated a few times as Lightning deprecates parameters/introduces parameters meaning users couldn't run the code out of the box unless they changed the configs manually.

It's the same with the CLI, if you want to save the config inside the repository.
python script.py --print_config > your_config.yaml will include all the current arguments for the trainer etc, if they are removed, the config will also become stale.
https://github.com/omni-us/jsonargparse/issues/79 helps by allowing us to specify metadata for this config, most importantly the version that generated it. This could be parsed later to improve the UX when the versions do not match.
https://github.com/omni-us/jsonargparse/issues/91 helps by printing only the arguments that have been customized, which reduces the chances of it breaking due to changes in other non-changed arguments.
Now, back to the main discussion.
I'd say, pick a few common or popular use cases, then we try implementing them with both options and see which is most natural/intuitive/simple, etc. This will also show which of the core components are inflexible.

Thanks for clarifying @carmocca!
Personally would like to give this is a go. I much prefer simpler packages, and I feel we could cut down complexity for elegancy and sustainability :)

Hey guys, I'm considering writing a tiny demo combining transformers with PL CLI. The Doc of Cli didn't explain too much about model inheriting. E.g. I can't use a config Dict anymore and have to write all the arguments in an explicit way. When it comes to multi-level inheriting, I got to write all the parameters manually. This is against simplicity. So is there any "best practice" or working-around of this problem?

hey @ElderWanng super late response.
Could you have a look at the latest code? Things have changed drastically, moving towards a class based approach :)

This is complete! For the next release, we'll be moving to a PyTorch Lightning focused library.


Add Initial Unit Tests

Add the following libraries:

mockito, network_image_mock and build_runner for mocking;
golden_toolkit for snapshot tests.

Add initial tests for a few files in the lib/common folder in order to validate the testing strategy and infrastructure.

@etabakov I added a few simple tests to validate how we want to test things. Particularly, I added the golden_toolkit library to make snapshot tests. WDYT about having such tests?

It's a dream come true for me - it will give so much peace of mind when deploying new versions!!
I'd lean on @vakrilov to weigh in on the best technical approach as I don't have too much experience with testing frameworks but definitely would like to invest in that.

It's exciting that we wouldn't have release the buggy version today if we had these tests :D

Yeah :) Let's add these first, and I can add more PRs with more tests.

Yes. Adding to CI first and then adding more tests sounds like the right sequence to me.


Remove requirement for server

To use the native WHMCS SSO functionality (to open a console on a VM), it requires a service to be assigned to a server, even with RequiresServer marked as false, as per https://developers.whmcs.com/provisioning-modules/meta-data-params/
Expected behaviour
Allow a user to SSO into a service without a WHMCS server assigned to it.
Actual behaviour
WHMCS errors with Server ID not found. when trying to SSO into a service without a server assigned to it.
Workaround
Setup an empty server for the Katapult module and set it as default, and ensure it is assigned to every Katapult service.

Case CORE-16422 now open with WHMCS.


0.1.11 fails to install new toolchain "rustup toolchain install 1.6.0"

I am receiving this error on x86_64 linux:
+ rustup toolchain install 1.6.0-x86_64-unknown-linux-gnu
info: syncing channel updates for '1.6.0-x86_64-unknown-linux-gnu'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.6.0.toml.sha256' to '/home/vagrant/.multirust/tmp/sszoa55jy97dkvg7_file
info: caused by: http request returned an unsuccessful status code: 404

I assume the channel listing is only for 1.8.0 (since that's the only fixed version with that toml file) but it is overeager to use it on versions lower than 1.8.0?

Sidenote: is there a direct download URL for older versions of rustup-init? I want to pin 0.1.10 directly in my Vagrantfile (for example) but I can't divine the URL for it, if it exists. e.g.:
https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-gnu/0.1.10/rustup-init

This is a regression. rustup should fall back to a different download strategy for versions prior to 1.8. This fallback is even in the test suite so I'm not sure what's going on.
@tcr3dr There are no release archives for rustup at present. Previous release binaries are discarded. I originally intended rustup to be simple enough to upgrade without worrying about it, but it's clear keeping past versions available would help with contingencies.

I'm looking into this and see what's going wrong. The server is returning error 403 but the fallback code only checks for 404. I'll have a patch up soon.


Move UseECM check for StartAdvertisement

Moved UseECM check above AppDelegate to ensure CommissioningWindowStatus is set before Commissioning Window is activated.
Problem

OnCommissioningWindowOpened runs before a window status is set. This may cause confusion as to which chip command triggered the commissioning window.

Change overview

Moved the UseECM check above OnCommissioningWindowOpened check.

Testing

No tests were conducted. Only a minor change to functions that are not dependent of each other.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Cliff Rawlings seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Set voltage references to report actual Battery and Solar voltages on Clima V4.b inputs

Modify the voltage reference used to scale the individual analogue inputs to account for the external resistor dividers on the Battery Voltage and Solar Voltage inputs on Clima V4.b PCB.

Amazing thx @doingnz


Added links and minor word modificaitons

@AErmie : Thanks for your contribution! The author, @rayne-wiselman, has been notified to review your proposed change.

Hi, thanks for this. There are some issues with these changes.

We only use sentence caps for headings.
We don't add multiple links to other pages/technologies. We place links to the other technologies in the specific table that calls these technologies out.

If you want to remove these two types of changes, we can submit. Thanks!

@AErmie : Thanks for your contribution! The author, @rayne-wiselman, has been notified to review your proposed change.

Thanks for the update. It's tricky to take these changes since some of them aren't required.  I will take the suggestions that should be updated and submit them in a different PR rather than go through them one by one in this PR. I'll report back to this PR with the details. Thanks!


Support for WINDOWS_UWP

Version Number and Operating System(s):
Latest from AssetStore or master (Unity folder)
Expected behavior:
Use the Forge asset as on Standalone/x64 platform
Actual behavior:
Unable to build UWP player.
Steps to reproduce:

Add Forge to project
Switch UWP platorm in Player settings
=> errors in Unity Console (Unfortunately, too many to write here)

I tried removing all WINDOWS_UWP code (to make the errors in Unity go away) but the build in the generated UWP project fails with:
1>FA35221275F8363D48A60E4904E1ADC5.obj : error LNK2019: unresolved external symbol ForgeConnect referenced in function TCPClientWebsockets_ForgeConnect_m77477C329CE2C5D77973A51A6D79C48B7EA4FF03
1>FA35221275F8363D48A60E4904E1ADC5.obj : error LNK2019: unresolved external symbol ForgeWrite referenced in function TCPClientWebsockets_ForgeWrite_m85DBCA448B5890BF8A8D2BC280E706BF2BDD261A
1>FA35221275F8363D48A60E4904E1ADC5.obj : error LNK2019: unresolved external symbol ForgeShiftDataRead referenced in function TCPClientWebsockets_ForgeShiftDataRead_m5BBDF3E6410CD77B9A91396C55B6F1C2D980D8CB
1>FA35221275F8363D48A60E4904E1ADC5.obj : error LNK2019: unresolved external symbol ForgeContainsData referenced in function TCPClientWebsockets_ForgeContainsData_m69EE45B1FE9AE2B353AF21E3CC6D2D08CCC65AFD
1>FA35221275F8363D48A60E4904E1ADC5.obj : error LNK2019: unresolved external symbol ForgeClose referenced in function TCPClientWebsockets_ForgeClose_m1F89438748E775FD81F81080058EF66626344AEC

https://github.com/BeardedManStudios/ForgeNetworkingRemastered/pull/292
UWP is sadly not supported anymore.

Though, remove the TCPClientWebsockets class to make it compile to IL2CPP. Looks like a few of those errors are because of that.

Closing, reopen if needed @eusebiu

Removing the class seems to generate compiler errors for other classes that depend on it, but replacing #if !UNITY_IOS with #if !UNITY_IOS && !WINDOWS_UWP got the linking to work for me on UWP.
Hope this helps someone.
@NFMynster Could you clarify if the UWP-specific code was dropped because perhaps it is not necessary (anymore) or because UWP is really not supported anymore by Forge and using Forge in UWP contexts will simply not work?
In case of the latter, it may be interesting to document it somewhere in the README or wiki; I was looking for this information before I started trying out Forge, and did not find this information - actually quite the contrary, since IL2CPP support is explicitly mentioned on the Asset Store, I (perhaps incorrectly) thought it would also support UWP, since the former is necessary for building UWP apps in Unity.

@tommrwatts I am in the same situation - I also have to support UWP/Hololens2 and I managed to make the build work by simply removing all code where WINDOWS_UWP was mentioned and basically use only the Mono/.NET code.
But, unfortunately, it is not working - I don't really understand why but I am not an expert on UWP.


Make write_initial_state default to true

📚 Documentation preview 📚: https://anemoi-inference--66.org.readthedocs.build/en/66/

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 75.00%. Comparing base (7ea4af7) to head (4ea4b3d).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop      #66   +/-   ##
========================================
  Coverage    75.00%   75.00%
========================================
  Files            1        1
  Lines            4        4
========================================
  Hits             3        3
  Misses           1        1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


How to get Specific Landmark From Face Mesh

Hi,
I need to get lips landmark from Face mesh.  I am getting 468 points and contours.
Q1: How to retrieve lips landmarks alone.
In code.
Addconnections method is used and they are drawing lines.  I need lips to be filled with specified color.
Q2: How to achieve color filling

Refer https://github.com/google/mediapipe/issues/1049#issuecomment-706858946


Add wrapper to create a slim read-only list

This should help with https://github.com/ppy/osu-framework/pull/3739 (it's a drop-in replacement). Supports List<T> for now, though in the future it could be upgraded to support more lists with minimal performance cost (~5us in the benchmarks).

 My benchmarks can't discern a cost involved in doing so, and it can always be upgraded into a reference if required.

Works well


Support for freestanding C++

I managed to get snitch to run properly on a freestanding C++ environment, namely the Sony PlayStation 1:

I encountered a few very minor issues however, and I'm reporting them here to see if we can create a working set of configuration settings to make this work. It'd be nice to have a SNITCH_FREESTANDING toggle that is.

Freestanding means no <cstring>, so no std::memmove and no std::memcpy. These could easily be defined away so the user can replace them at compilation time with macros indicating the replacement of their choice, such as __builtin_memmove and __builtin_memcpy when using gcc. These occur in snitch_append.cpp and snitch_string_utility.cpp.
Freestanding means no <cstdio>, so no snitch_console.cpp nor snitch_file.cpp. My opinion for these is to simply disable most of the implementation details in freestanding mode, and let the user provide alternative implementations for them, in the form of simple C-style externs, such as void snitch_freestanding_write_console(const char * data, size_t len), void * snitch_freestanding_open_for_write(const char * fname), void snitch_freestanding_write(void *, const char * data, size_t len), and void snitch_freestanding_close(void *), to simplify interoperability.

I can issue a pull request to address these, if that's an acceptable design.

I'm currently traveling so my response will be brief, but I just wanted to say thank you for sharing this. This is cool!
I have no experience with freestanding environments, but what you propose makes sense in spirit.
For memcpy/memmove, could an alternative be to simply handwrite the copy algorithm if SNITCH_FREESTANDING (to be added) is set to 1? I assume the compiler can detect the pattern and replace it with the appropriate builtin (with optimisations enabled...). Would need testing. That would require less setup on the user side. Alternatively, is <algorithm> available?
For <cstdio>, your proposal makes sense. Though, the mechanism we have been using so far is to use function_ref (essentially a function pointer that can't be null and also works with member functions) rather than an extern declaration. This is already done to customise the function to use for console output, in particular. Could be done for file output too.
Though in the freestanding case there is no suitable default, as far as I understand, so we'd have to initialise these refs to "unimplemented" functions (which terminate on call), and that is not very nice.
We could instead initialise the functions refs to undefined extern declarations, which comes back to your proposal but with the added ability to also use member functions if needed. The downside is then that the functions must be defined, even when not used, which is also not very nice.
I'm reluctant to introduce macros other than 0/1 and numerical values, to keep things simple.

I'm currently traveling so my response will be brief, but I just wanted to say thank you for sharing this. This is cool!

Oh, please do enjoy your travel, I'm in no rush at all. Thank you for taking the time to reply and your detailed feedback :)

I have no experience with freestanding environments, but what you propose makes sense in spirit.

If I'm being perfectly honest, I tried to go for the fastest evaluation path, in a fail fast manner, and I didn't necessarily spend a huge lot of time looking at the implementation details for what the best solution could be. It turns out your library is exceedingly viable for freestanding environments with no FPU, RTC, or filesystem, and while a basic demo test creates a 385kB binary which, for a machine where you have a bit less than 2MB available, is a lot, it's still acceptable for only a test.

For memcpy/memmove, could an alternative be to simply handwrite the copy algorithm if SNITCH_FREESTANDING (to be added) is set to 1? I assume the compiler can detect the pattern and replace it with the appropriate builtin (with optimisations enabled...). Would need testing. That would require less setup on the user side. Alternatively, is <algorithm> available?

<algorithm> is totally available, yes, and std::copy is a thing, as I guess you were probably thinking of it. I guess the main difference with std::memcpy and std::memmove is it won't allow nullifying some UB caused by data conversion and reinterpretation, but while I only glanced at your code, it seems you're not relying on that at all, so it could be a perfectly viable change instead. I haven't looked deeply on why you need std::memmove, but the copy direction is only a small implementation detail on top of std::copy itself.

For <cstdio>, your proposal makes sense. Though, the mechanism we have been using so far is to use function_ref (essentially a function pointer that can't be null and also works with member functions) rather than an extern declaration. This is already done to customise the function to use for console output, in particular. Could be done for file output too.
Though in the freestanding case there is no suitable default, as far as I understand, so we'd have to initialise these refs to "unimplemented" functions (which terminate on call), and that is not very nice.

I see it now, yes. This works for me. I just haven't seen how it works just yet, but that's pretty much irrelevant: it's there, and it's usable. If you don't like the idea of having a default which crashes, these could also just be no-ops. They'd just... do nothing. I'm a bit more partial to an std::terminate myself, as displaying nothing feels like a more frustrating debugging experience for the user, in case they forgot to set the console output.

We could instead initialise the functions refs to undefined extern declarations, which comes back to your proposal but with the added ability to also use member functions if needed. The downside is then that the functions must be defined, even when not used, which is also not very nice.
I'm reluctant to introduce macros other than 0/1 and numerical values, to keep things simple.

As I didn't dig enough, I haven't noticed you don't have any other macros beyond 0/1 and numerical values, indeed, and I totally agree with you on that one.
Couple of remarks, then: we could have a new SNITCH_FREESTANDING toggle, which then cascades into two other toggles: SNITCH_WITH_CONSOLE and SNITCH_WITH_FILE, which would each bump out the current implementations for console and files into no-ops or std::terminate. In fact, I technically would only need a console output myself, as I don't intend on setting the -out option, nor using any generator. For now. Which simplifies the implementation details overall.
In addition to the usage of std::copy, I could quickly send out a pull request. I'm not 100% sure how we could proceed for the unit testing of all that however, but spinning up a freestanding environment takes a few minutes of gcc compilation time in a github workflow, and I have scripts ready for the case of the PlayStation 1. It's just that running these tests can be a bit more challenging, at face value. And I haven't tried Linux-on-Linux freestanding gcc yet, and I'm not completely sure on how reliable that'd be as a test.
Also, notable: as I said, I went for the fastest path to trying things out, and I used the following toggles, different from your defaults. Not totally sure this is relevant, but it's something to consider:

SNITCH_WITH_EXCEPTIONS 0
SNITCH_WITH_TIMINGS 0
SNITCH_APPEND_TO_CHARS 0
SNITCH_WITH_ALL_REPORTERS 0
SNITCH_WITH_MULTITHREADING 0

It turns out your library is exceedingly viable for freestanding environments with no FPU, RTC, or filesystem, and while a basic demo test creates a 385kB binary which, for a machine where you have a bit less than 2MB available, is a lot, it's still acceptable for only a test.

That's great to hear! I haven't really made a concerted effort to reduce binary size, although it was a consideration when writing the float-to-string implementation. The state-of-the-art algorithms do have much bigger tables, which seemed to me were not worth the cost.
When you say there is no FPU, does that mean that floating point calculations are still possible, but emulated in software? Does this have any impact on C++ code, other than performance considerations? I'm asking mostly to see if it would make sense to add a compile-time toggle to remove the float-to-string code and all you to save more space.
Another question on binary size: have you tweaked any of the SNITCH_MAX_* options? I expect the default SNITCH_MAX_TEST_CASES of 5000 is probably a bit much for embedded platforms... Reducing this and others might help you shrink the binary footprint further.
Lastly, I'm dropping this here because I think it's very relevant; there was a recent blog article from the author of fmtlib, discussing how to reduce binary size. We don't use fmtlib nor std::format, but the article has a few cool tricks for investigating this kinds of issues, that may be worth looking into.
It inspired me to try compiling snitch without linking to any C or C++ runtime to see what we actually depend on. Oddly, even with -fno-exceptions, much of the dependence on the C++ runtime was exception-related stuff in std::optional and std::variant. Apparently, even with -fno-exceptions, libstdc++ does let these classes throw... Maybe I need to compile against an actual freestanding STL.

I haven't looked deeply on why you need std::memmove

The append() function arguments can alias each other:
snitch::small_string<128> s;
append(s, "abc");
append(s, s); // aliasing

I initially thought this could cause UB because of overlapping memory ranges, but actually append() always copies past the end of the first argument, which in principle points to unclaimed memory. So I think by construction the ranges can never overlap in normal use :thinking:. But they can be made to, with a bit of unsanitary code:
snitch::small_string<128> s;
append(s, "abc");
std::string_view old_s = s;
s.clear();
append(s, old_s); // now the ranges actually overlap

Since the small_string never re-allocates, and since char is a trivial type, old_s actually points to valid memory. So with std::memmove this code doesn't cause UB today, but it would if we had used std::memcpy or std::copy. But it's asking for trouble... We don't rely on this behaviour in snitch, so perhaps std::copy would be fine.
The only reason I didn't use std::copy was that I was initially weary of including <algorithm>, which is a notably heavy header, and I feared for compilation time. But it's in in a *.cpp file, and other files now include <algorithm> too, so I don't think that's a valid worry anymore.

I see it now, yes. This works for me. I just haven't seen how it works just yet, but that's pretty much irrelevant: it's there, and it's usable.

Here's how that would work for the console output:
// Define you own printing function.
void my_console_print(std::string_view message) noexcept {
    // Do whatever you need to do to print the provided characters...
}

// Then in main():
int main(int argc, const char* argv[]) {
    // Replace the print function used for the command-line interface and debug messages:
    snitch::cli::console_print = &my_console_print;
    // Replace the print function used for test reporting:
    snitch::tests.print_callback = &my_console_print;
}

For files it's not possible today, but we could use a similar mechanism to override the open/write/close functions. The difficulty is managing the file state (e.g. FILE* pointer in C, or std::ofstream object in C++). We have already solved this for reporters; the idea is to use an inplace_any<max_file_object_size> (with a configurable max_file_object_size) to store this state. Then the three functions become (here implemented using std::ofstream):
void file_open(inplace_any<max_file_object_size>& storage, std::string_view path) {
    storage.emplace<std::ofstream>(path);
}

void file_write(inplace_any<max_file_object_size>& storage, std::string_view message) {
    storage.get<std::ofstream>() << message;
}

void file_close(inplace_any<max_file_object_size>& storage) {
    storage.reset();
}

(could be made a little nicer if we introduced a type-erased inplace_any_view so that we don't have to repeat the size in the template parameter...)

I'm a bit more partial to an std::terminate myself, as displaying nothing feels like a more frustrating debugging experience for the user, in case they forgot to set the console output.

I agree!

Couple of remarks, then: we could have a new SNITCH_FREESTANDING toggle, which then cascades into two other toggles: SNITCH_WITH_CONSOLE and SNITCH_WITH_FILE, which would each bump out the current implementations for console and files into no-ops or std::terminate.

As above, I think we'd go for std::terminate. Otherwise, perhaps some bike-shedding on names: rather than split consoles vs files, we could simply have a single SNITCH_WITH_STDIO. I think it's unlikely that you'd have console but not files, or vice versa.
I like the idea of adding SNITCH_FREESTANDING, as that could cascade into other defines (no exceptions, no threading, etc.). There are a few things to decide, such as what happens if someone sets SNITCH_FREESTANDING=ON and SNITCH_WITH_STDIO=ON (IMO: the freestanding toggle takes precedence, and turns stdio off).

I'm not 100% sure how we could proceed for the unit testing of all that however, ...

I think spinning up a Playstation 1 environment might be overkill. Perhaps a simpler approach would be to set SNITCH_FREESTANDING=ON, define a "do nothing" function for the console output, and configure snitch to not link to libc or libstdc++ (the fmtlib articles shows how to do that). In principle this should compile and run. Though, as above, we may need to compile against a freestanding STL. It seems GCC has a -ffreestanding toggle that enables this, but this may require GCC 13 to work properly.
As to actually checking that the run is successful without having console output, we could simply check the return code of the test application (GitHub actions already do that automatically). That will be annoying to debug when the tests fails though... The nicer alternative would be to build our own "standard library", creating a shared library that only exposes wrappers around the <cstdio> functions that we need, and configure snitch to use that. That should mimic a free-standing environment (does it?) and still give us console output to debug problems when they occur.

When you say there is no FPU, does that mean that floating point calculations are still possible, but emulated in software? Does this have any impact on C++ code, other than performance considerations? I'm asking mostly to see if it would make sense to add a compile-time toggle to remove the float-to-string code and all you to save more space.

Yes, the compiler will have to emit soft float implementation, which is fairly bloated, and slow. It should be considered the same as with exceptions or rtti. However, I didn't really see any float usage straight off? Wasn't your conversion code working at compilation time? Or I didn't necessarily spend too much time investigating what I was looking at.

Another question on binary size: have you tweaked any of the SNITCH_MAX_* options? I expect the default SNITCH_MAX_TEST_CASES of 5000 is probably a bit much for embedded platforms... Reducing this and others might help you shrink the binary footprint further.

Yes, this definitely helps. But the default values are very viable still, which is good for entry-level users.

Here's how that would work for the console output:
// Define you own printing function.
void my_console_print(std::string_view message) noexcept {
    // Do whatever you need to do to print the provided characters...
}

// Then in main():
int main(int argc, const char* argv[]) {
    // Replace the print function used for the command-line interface and debug messages:
    snitch::cli::console_print = &my_console_print;
    // Replace the print function used for test reporting:
    snitch::tests.print_callback = &my_console_print;
}

Right, I see now you can manually define main; seen the proper documentation. It makes sense.

Otherwise, perhaps some bike-shedding on names: rather than split consoles vs files, we could simply have a single SNITCH_WITH_STDIO. I think it's unlikely that you'd have console but not files, or vice versa.

Wellll... it's actually not that odd. Console usually would be a tty interface through uart for instance, while files require an actual filesystem working in the backend, which isn't really a guarantee. The PS1 has a standard tty system in its kernel, but no default writable filesystem (it's cd-rom based). It's similar to other embedded environment too, not just the PS1.

I think spinning up a Playstation 1 environment might be overkill. Perhaps a simpler approach would be to set SNITCH_FREESTANDING=ON, define a "do nothing" function for the console output, and configure snitch to not link to libc or libstdc++ (the fmtlib articles shows how to do that). In principle this should compile and run. Though, as above, we may need to compile against a freestanding STL. It seems GCC has a -ffreestanding toggle that enables this, but this may require GCC 13 to work properly.
As to actually checking that the run is successful without having console output, we could simply check the return code of the test application (GitHub actions already do that automatically). That will be annoying to debug when the tests fails though... The nicer alternative would be to build our own "standard library", creating a shared library that only exposes wrappers around the <cstdio> functions that we need, and configure snitch to use that. That should mimic a free-standing environment (does it?) and still give us console output to debug problems when they occur.

So the problem is that you can't trust libstdc++ to honor the freestanding toggle. I was the one to file bugs against the gcc project in the first place which got freestanding mode in libstdc++ to even build and work not long ago. And the result is that when you compile gcc/g++, you specify you want to build a libstdc++ which is freestanding. The resulting libstdc++ will be smaller and have less features.
Here's the gist: if you use the freestanding toggle, it'll affect the codegen, but not much the precompiled libstdc++ code itself. And a freestanding libstdc++ which has been built from source as freestanding will have missing headers like cstdio which won't get installed. In a normal gcc environment, you'd be able to compile code against cstdio for instance, even with the freestanding compiler toggle, simply because the header is present on the filesystem.
My point being you wouldn't safely be able to rely on the freestanding alone to ensure the code properly compiles in a fully freestanding environment, and you could get false positive results.

However, I didn't really see any float usage straight off? Wasn't your conversion code working at compilation time? Or I didn't necessarily spend too much time investigating what I was looking at.

It's used to implement compile-time float-to-string serialization yes, but also for run-time serialization if SNITCH_APPEND_TO_CHARS is 0 (which you do set). Though the tables used by the conversion functions make up only 1520 bytes, which is far less than most modern float-to-string algorithms (e.g. ryu's tables are about 100kB). And if you disable timings, and you don't use floats in your own test code, I don't think floats are used anywhere in the program so this might all end up being compiled-out anyway.

Wellll... it's actually not that odd. Console usually would be a tty interface through uart for instance, while files require an actual filesystem working in the backend, which isn't really a guarantee. The PS1 has a standard tty system in its kernel, but no default writable filesystem (it's cd-rom based). It's similar to other embedded environment too, not just the PS1.

Interesting! OK, makes sense to keep them separate then.

My point being you wouldn't safely be able to rely on the freestanding alone to ensure the code properly compiles in a fully freestanding environment, and you could get false positive results.

I understand, that's unfortunate. I think the simplest alternative that is left would be to still compile against a non-freestanding STL, but check that the link-time dependencies don't include anything that would not be available in a freestanding environment. That sounds doable.

For example, with the build options you selected above, linking the resulting libsnitch.a to a simple test application like so:
#include <snitch/snitch.hpp>

TEST_CASE("test") {
    CHECK(1 == 2);
}

And calling objdump -T test, we get:
test:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.34) __libc_start_main
0000000000000000  w   D  *UND*<PHONE_NUMBER>000000  Base        __gmon_start__
0000000000000000  w   D  *UND*<PHONE_NUMBER>000000  Base        _ITM_deregisterTMCloneTable
0000000000000000  w   D  *UND*<PHONE_NUMBER>000000  Base        _ITM_registerTMCloneTable
0000000000000000  w   DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) __cxa_finalize
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) strlen
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (CXXABI_1.3) __gxx_personality_v0
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.4)  __stack_chk_fail
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) abort
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) memcmp
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) memchr
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBCXX_3.4.20) _ZSt24__throw_out_of_range_fmtPKcz
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) memmove
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) memset
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) fwrite
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBCXX_3.4) _ZSt9terminatev
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) fopen
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) fclose
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) fflush
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.14) memcpy
0000000000000000      DF *UND*<PHONE_NUMBER>000000 (GLIBC_2.2.5) __cxa_atexit
0000000000032b44  w   DF .text<PHONE_NUMBER>000025  Base        _ZNSt11char_traitsIcE2eqERKcS2_
00000000000faa18 g    DO .bss<PHONE_NUMBER>000008 (GLIBC_2.2.5) stdout

We can add a test that checks this output against an allow-list; any entry not in the allow-list would flag a test failure. My understanding is that we can keep all of these except:

fopen / fclose / fflush -> will be gone with SNITCH_WITH_FILE=off
stdout -> will be done with SNITCH_WITH_CONSOLE=off
fwrite -> currently used both for writing to file and the console... do you have that one? Or will we need to move to printf() for the console?
memcpy / memmove / memchr / strlen / memcmp -> can be replaced by C++ algorithms (I worry that a non-freestanding C++ STL will use these under the hood though...) or hand-rolled
_ZSt24__throw_out_of_range_fmtPKcz -> not sure what is introducing that one, but probably std::optional or std::variant

In particular I assume these ones are OK, but let me know if not:

abort, not sure where that one comes from
_ZSt9terminatev (std::terminate()), used all over the place in snitch
_ZNSt11char_traitsIcE2eqERKcS2_ (std::char_traits<char>::eq(char const&, char const&)), probably from std::string_view


Doesn't work with Arduino Nano Every

This example code works with the Arduino Uno:
#include <Tone.h>

Tone tone1;

void setup()
{
  tone1.begin(13);
  tone1.play(NOTE_A4);
}

void loop()
{
}

But I get this error below with the Arduino Nano Every (probably because of differences between ATmega328P and the ATmega4809).
Tone.cpp:560:37: error: 'OCIE2A' was not declared in this scope
       returnvalue = (TIMSK2 & (1 << OCIE2A));

Any help?

I'd like to see this addition as well - the timers used between the 4809 and the 328 are of course very different, but the Nano Every is a great board and is also cheaper, a lot of universities are using these now instead of the Nano so it would be great to have this library work with it


Hide the error message when deleting temp OSS-Fuzz dir

Hiding the error message details because:

The error is expected in all local experiments: OSS-Fuzz dir (/tmp/<tmp_dir>/build/work) contains the files copied from the docker container, which requires root privilege to delete.
The error is benign: Temp dir is in /tmp and will be automatically deleted by OS.
The error is hard to fix: Fixing this requires using sudo or docker, which adds too much unnecessary effort.

We did not have this problem before because we used pass:
https://github.com/google/oss-fuzz-gen/commit/9885ae16b9f12686b504eecd94b35641f27c608a#diff-c265089165acff162907dbef5c22b1f966e638b1e750bee28c5a548262464fc3L40
Now, we also log a warning in case this happens elsewhere.

I'm not sure why it's worth removing the exception but this change dosn't seem to break anything to me.

Sorry, I should provide a bit more context:
Previously, we held the temp dir of OSS-Fuzz in a local variable of a function. Hence, the dir may be recycled during the experiment. The good thing is that the long and annoying error message will not pop at the end of experiments, where we print out the results. The bad thing is that some trials failed to find the OSS-Fuzz dir during the experiment and break.
I fixed it by holding it in a global variable. This makes the temp last till the end, but the long error messages will also show up at the end, which becomes annoying.
I should probably also remove the other e, for the same reason.


Disables pinch zoom on Ipad

Hi I am using this plugin as Kinetic.js (html 5 canvas library) disables scroll when tyring to scroll over stage objects. This plugin seems to resolve this issue.
However I notice that it also disable pinch zoom. Is this the correct behavour?
I am applying it as to the body and html elements like so:
$('html,body').kinetic();

This library is still useful on iOS, even after all this time, as -webkit-overflow-scrolling: touch. If it helps anyone, you can enable pinch-to-zoom by doing:
filterTarget: function( target, e ) {
    if ( e.originalEvent.touches !== undefined && e.originalEvent.touches.length > 1 ) {
        return false;
    }
}


Add system specs for new fields

I rule and have completed some work on Case Manager that's ready for review!
Adds system tests for new form fields
It relates to the following issue #s:

Fixes #1828

thank you very much @valeriecodes !


Minor changes

The StringVal struct methods are named properly to reflect their functionality and get_value() is changed to meet expected scenarios. Also include some minor changes

@agentjill
Thank you for your continued support.
We will merge them.


Make some commands async

Some comands use libraries that doesn't work good in an async enviroment.

[ ] j!yt needs to use aiohttp
[x] j!wolframalpha uses wolframalpha library, which uses urllib
[x] j!temperature uses pyowm which looks async, but needs further testing

https://github.com/lkmnds/jose/commit/f4c4cb3fb3fd8f87586ec22578285116bd8e5a8a fixes j!wolframalpha and j!temperature by using loop.run_in_executor.

https://github.com/lkmnds/jose/commit/08c866c5db550f4357002738468fab0b9f78a13f for j!yt


DataGrid server-side row selection

Fix: https://pastebin.com/ZFDwQgWN

Updated fix: https://pastebin.com/FuQTQe1P

Fixed by #26


URL Shortener

URL Shortener
Close #1037
Checklist:

[x] I have mentioned the issue number in my Pull Request.
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have created a helpful and easy to understand README.md

@MohitGupta121 Please review and accept the PR under GSSoC'23

@MohitGupta121 I have added the screen shot already should I add screen recording too?


Covid deaths as rate by 100k

Built on work that Te did. Get 2020 population for each race from Erica's numbers. Then get the rate per race. Finally then sum up the cases per race and population by race to calculate rate for all people in a geography.

Yep you're 100% right, that's an important refactor. Wrote an issue and assigned us both


feat: implement VGG13, VGG16 and VGG19

This PR is a follow up to https://github.com/huggingface/candle/pull/1209 and implemented VGG13/16/19 and relies on hf-hub for the models

Thanks for the PR, I'm pushed a couple tweaks on top of it. Most of them are cosmetic and to make the code a bit more rust idiomatic and avoid intermediary clone() that are not required. A change that is a bit more important is that the weights were reloaded each time the final layer was applied as this happened in the closure, this should now be fixed.


lfs: add support for Git SSH URLs

I've added support for Git SSH URLs to the Git LFS client. Resolves #288.
I've refactored the LFSClient class a little to have the common code in LFSClient and implement logic specific to Git HTTP URLs and Git SSH URLs in subclasses. The LFSClient class is now abstract with one abstract method _get_auth_header(upload: bool): dict which gets implemented by the private subclasses _HTTPLFSClient and _SSHLFSClient. _HTTPLFSClient retrieves credentials as before using the credential helper, but instead of using AIOHTTP's auth argument, it creates the basic auth header manually using BasicAuth(...).encode() which gets passed along with other headers. _SSHLFSClient retrieves credentials using ssh git@<host> <project_path> upload|download (as documented in the Git LFS SSH authentication docs) and takes the header dict from its response, which contains authentication data.
Some questions regarding this implementation:

Is using _get_ssh_vendor from scmrepo.git.backend.dulwich the right way to create an SSH client?
Any suggestions about testing? I don't feel too comfortable adding this feature without tests, but I'm not sure what the best approach to testing would be. Perhaps using a Git LFS test server? I've manually tested both Git SSH URLs and Git HTTP URLs of public and private projects, all scenarios are working for me.

Is using _get_ssh_vendor from scmrepo.git.backend.dulwich the right way to create an SSH client?

Ideally I think we should be using asyncssh directly so that this is independent of any particular git backend. I'm not sure how the CLI LFS client works with things like GIT_SSH_COMMAND, so it may actually be necessary to use dulwich if we do need to support falling back to a user-provided SSH shell command.
Either way, using dulwich here should be fine for now.

Any suggestions about testing? I don't feel too comfortable adding this feature without tests, but I'm not sure what the best approach to testing would be. Perhaps using a Git LFS test server? I've manually tested both Git SSH URLs and Git HTTP URLs of public and private projects, all scenarios are working for me.

I think testing this will really require setting up a docker container that is running a git + LFS + SSH server instance. We do already have a minimal setup for testing git over SSH, you can take a look at tests/conftest.py and tests/docker-compose.yml to get an idea of how that setup works.

In general this PR LGTM (other than the testing issue), but I should note that my understanding is that this use of LFS auth over SSH is not widely used, and the alternative (HTTPS auth) works for github/gitlab as well as huggingface. So I'm not sure this feature is something that needs to be prioritized.
@sisp is your team actively using LFS auth over SSH?
(Full disclosure, I'm leaving the DVC team at the end of this week, and at least in the short term, the rest of the team won't have the bandwidth to work on or review changes in the scmrepo LFS client other than bugs where something is completely broken with github/gitlab/huggingface)

Thanks for your comprehensive feedback, @pmrowla! :pray: First of all, I'm sorry to hear you're leaving the DVC team. Your work on LFS support here – and thus also for DVC – has been fantastic and triggered an idea for a partial solution related to my struggles with tightly integrating DVC and GitLab (https://github.com/iterative/dvc-http/issues/50#issuecomment-1605291458, https://github.com/iterative/dvc-http/pull/56, gitlab.com:gitlab-org/gitlab#413612) by simply using Git LFS for data/artifacts storage instead of a DVC remote. This is feasible on our corporate self-hosted GitLab instance, as it has a very generous LFS storage size limit, so storing significant data and artifacts is no problem. With DVC's LFS support, we can, e.g., import data managed via LFS from non-DVC repos and yet benefit from DVC's other features. That said, LFS is only a substitute for the regular cache but not the run-cache.
Since there's little time left before you leave, I'm inclined to leave the PR as is and look into testing in a follow-up. Also, I've looked into refactoring the current approach to using authenticator classes. I realized that not only retrieving credentials for the LFS Batch API is different for HTTP and SSH, but also constructing the LFS server URL from the Git URL is different. An authenticator class should not cover the URL construction, as that's a different concern, but authentication and URL construction are closely related. For this reason, I think we should keep the inheritance-based approach as is.
We use Git over SSH on our GitLab instance by default. For us, 2FA is always enabled, and then:

When 2FA is enabled, you can’t use your password to authenticate with Git over HTTPS or the GitLab API. You can use a personal access token instead.
— https://docs.gitlab.com/ee/user/profile/account/two_factor_authentication.html#use-personal-access-tokens-with-two-factor-authentication

The same applies to GitHub. In fact, I use 2FA and Git over SSH also on github.com and gitlab.com because it's secure and convenient. Git over HTTP only works nicely without 2FA or with public projects IMO, so I think supporting LFS for Git SSH URLs is a high priority in corporate environments, e.g. when not only consuming public LFS-managed files but also internal ones.


Updated homepage in cabal file.

Update the homepage in the .cabal file to point to https://github.com/WraithM/opaleye-trans instead of https://github.com/tomjaguarpaw/haskell-opaleye

I don't necessarily agree with this change. I have the link to this repo in the source-repository section. Is there a reason why you think it should be this page as opposed to tomjaguarpaw's repo, which is sorta the homepage for opaleye generally.

@WraithM Sorry for not explaning this better in my initial commit!
I had assumed that the homepage section in the cabal file for this package was incorrect.  I had (mistakenly) thought that you just copied the .cabal file from opaleye and forgot the change the homepage setting.
As an end user, it was confusing as to why the homepage for the opaleye-trans package points to opaleye, when there is no relationship between opaleye and opaleye-trans (I think?).  I could understand if opaleye-trans was officially supported by opaleye (but I don't think it is?).
IMO, I think the homepage should point to this repository, and then the description or synopsis should have a link to the opaleye repository.  As an end-user, I think that would make more sense.
Of course, this is completely just bike-shedding, so feel free to ignore it and close this PR :-)

Sure, you convinced me. I'm not really affiliated tomjaguarpaw.


Using SimpleTcp in UWP

Hello,
I wanted to use simpleTcp in a uwp project but got an error. Should I do something different when using Uwp?
My Code
SimpleTcpClient = new SimpleTcpClient(Ip, Port, false, null, null);
SimpleTcpClient.Connect();
   if (SimpleTcpClient.IsConnected)
   {
       SimpleTcpClient.Events.DataReceived += Events_DataReceived;
       SimpleTcpClient.Events.Connected += Events_Connected;
       SimpleTcpClient.Events.Disconnected += Events_Disconnected;
   }

Error
System.Net.Internals.SocketExceptionFactory.ExtendedSocketException: 'An attempt was made to access a socket in a way forbidden by its access permissions <IP_ADDRESS>:23'

I solved the problem. I did not give the necessary permissions in the access permissions of the application. There may be others who encounter the same problem in the future. The roadmap is as follows.
Package.appxmanifest -> Capabilities -> Private Networks (Client & Server) Check


Java reflection: return empty array when no reflection data available

getDeclaredMethods() and getDeclaredFields() call fails when no reflection data available. In this case process and return an empty array.

This should not work this way.
First, set of classes that are ever accessed via getDeclaredMethods/getDeclaredFields, is automatically computed (here, so there should not be the case when 'reflection is not available'. Metadata is always available. The mechanism described in documentation only affects the ability to call method. If you encounter case when you access getDeclaredMethods or getDeclaredFields and get weird exception, this is affected by bug in detection of set of classes with metadata available. Can you please provide an example that reproduces this issue?
Second, if at some point a new mechanism, that allows to tune the set of classes with metadata, is introduced, I believe, in this case attempt to call getDeclaredMethods or getDeclaredFields should throw SecurityException.


Add a confirm option to CLI vm create commands which cause downtime

Reference: Feature 4947992: [Linux E2E] Add a confirm option to CLI vm create commands which cause downtime
Description:
"If you az vm create with a size, say DS1_v3, and then repeat that same command with DS2_v3 or another size, it changes the VM size. (I was trying to create many instances and forgot to change the -n parameter)
It might be by design, but I wonder if we should warn customers and ask them to add a --yes or something, since it implies downtime. We get that info notice in the portal, not sure if CLI should emit some form of warning.
The RP probably can't do anything unless the "create or update" functionality is split in destructive vs. non-destructive updates. In EC2, there's a "modify instance attribute" method and the VM size can only be changed with a stopped VM."
We already have “az vm resize” so adding some kind of update parameter to destructive creates would make sense.

@qwordy  please take a look and comment.

It's by design. It's called idempotence.
https://blogs.msdn.microsoft.com/cloud_solution_architect/2017/08/09/understanding-idempotence-and-scope-in-azure-resource-manager-templates/
But we'll discuss your advice.

I disagree that this behavior is a result of idempotent design. I think the concepts are orthogonal.
I don't think anyone is disagreeing that issuing a second az vm create command with different parameters will result in a scoped deployment that is sufficiently different from the original one to mutate the state of the resource.
The point of this issue is that, unlike the example covered in the MSDN article which isn't disruptive of the VM operation, changing the VM size automatically reboots the instance with no warning to the user.
While the Compute RP could change to accommodate this behavior (as it is done in other cloud providers) that's not what this issue is about. This is about issuing sufficient warning in azure-cli that such a specific change in the deployment (VM size) will reboot the instance.
Unlike an az vm restart operation, where the verb pretty much indicates what's going to happen, a user might not expect that reissuing az vm create (possibly because they forgot to change the VM name) will restart an existing VM.

That makes sense. Changing the VM size automatically reboots the instance with no warning to the user. It's unexpected for some people. But it's hard to make a breaking change which adds a confirmation. I can add a warning first. And I will raise this issue to the team and have a discussion.
It's general problem of Azure CLI. We allow creating an existing resource again by create command, while the actual behavior is updating the existing resource.


All rebuild when increment counter

All (Scaffold)rebuild when increment counterX
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.3.9, on macOS 12.5 21G72 darwin-arm, locale fr-FR)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.3)
[✓] VS Code (version 1.71.2)
import 'package:equatable/equatable.dart';
import 'package:flutter_riverpod/flutter_riverpod.dart';

class CounterNotifier extends StateNotifier<CounterState> {
  CounterNotifier() : super(const CounterState());

  increment() {
    int count = state.counterX;
    state = state.copyWith(counterX: count + 1);
  }
}

class CounterState extends Equatable {
  final int counterX;
  final int counterY;

  const CounterState({
    this.counterX = 0,
    this.counterY = 50,
  });

  @override
  List<Object> get props => [counterX, counterY];

  CounterState copyWith({
    int? counterX,
    int? counterY,
  }) {
    return CounterState(
      counterX: counterX ?? this.counterX,
      counterY: counterY ?? this.counterY,
    );
  }
}

final counterNotifierProvider =
    StateNotifierProvider<CounterNotifier, CounterState>((ref) {
  return CounterNotifier();
});

import 'package:flutter/material.dart';
import 'package:flutter_riverpod/flutter_riverpod.dart';
import 'package:labo/counter_notifier.dart';

void main() {
  runApp(const ProviderScope(child: MyApp()));
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        // This is the theme of your application.
        //
        // Try running your application with "flutter run". You'll see the
        // application has a blue toolbar. Then, without quitting the app, try
        // changing the primarySwatch below to Colors.green and then invoke
        // "hot reload" (press "r" in the console where you ran "flutter run",
        // or simply save your changes to "hot reload" in a Flutter IDE).
        // Notice that the counter didn't reset back to zero; the application
        // is not restarted.
        primarySwatch: Colors.blue,
      ),
      home: const MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends ConsumerStatefulWidget {
  const MyHomePage({super.key, required this.title});

  // This widget is the home page of your application. It is stateful, meaning
  // that it has a State object (defined below) that contains fields that affect
  // how it looks.

  // This class is the configuration for the state. It holds the values (in this
  // case the title) provided by the parent (in this case the App widget) and
  // used by the build method of the State. Fields in a Widget subclass are
  // always marked "final".

  final String title;

  @override
  ConsumerState<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends ConsumerState<MyHomePage> {
  void _incrementCounter() {
    ref.read(counterNotifierProvider.notifier).increment();
  }

  @override
  Widget build(BuildContext context) {
    final counterX =
        ref.watch(counterNotifierProvider.select((value) => value.counterX));
    final counterY =
        ref.watch(counterNotifierProvider.select((value) => value.counterY));

    return Scaffold(
      appBar: AppBar(
        // Here we take the value from the MyHomePage object that was created by
        // the App.build method, and use it to set our appbar title.
        title: Text("App"),
      ),
      body: Center(
        child: Column(
          children: [
            Text("CounterY 250 : $counterY"),
            Text("CounterX: $counterX"),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _incrementCounter,
        tooltip: 'Increment',
        child: const Icon(Icons.add),
      ),
    );
  }
}

That's normal. MyHomePage rebuilt.
MyApp did not, so all is working as expected
If you want to rebuild only the text, you can use Consumer and encapsulate only the Text


DeviceSegmentedReduce out-of-memory error

Hi Duane,
We encountered a memory error when using DeviceSegmentedReduce::Reduce, specifically, 	cudaErrorMemoryAllocation error in the reduce kernel DeviceSegmentedReduceKernel().  The error only occurs when we compile with -g -G debug flags, and when there are more than 4 floats in a struct. I think there is a bug somewhere in the cub, but I used gdb and it does not give me much info.
Here is the step to reproduce the error:

compile the code with:
nvcc -g -G  -c -std=c++11 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70  -I/cm/extra/apps/CUDA.linux86-64/9.1.85_387.26/include -I/cm/extra/apps/cuDNN/9.1-v7.0.5/include -I/cm/extra/apps/CUDA.linux86-64/9.1.85_387.26/samples/common/inc -I/home/dorisp/TarBalls/cub-1.8.0 -I./ -o segmentedReduce.o segmentedReduce.cu
nvcc -g -G  -std=c++11 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70  -o segmentedReduce segmentedReduce.o -L/cm/extra/apps/CUDA.linux86-64/9.1.85_387.26/lib64

Then run:
./segmentedReduce
...
temp_storage_bytes = 1
CUDA error at segmentedReduce.cu:179 code=2(cudaErrorMemoryAllocation) "cub::DeviceSegmentedReduce::Reduce( d_temp_storage, temp_storage_bytes, d_in, d_out, num_segments, d_offsets, d_offsets + 1, reduceOp, initVal )"

compiler the code without -g -G, then it runs fine
nvcc  -c -std=c++11 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70  -I/cm/extra/apps/CUDA.linux86-64/9.1.85_387.26/include -I/cm/extra/apps/cuDNN/9.1-v7.0.5/include -I/cm/extra/apps/CUDA.linux86-64/9.1.85_387.26/samples/common/inc -I/home/dorisp/TarBalls/cub-1.8.0 -I./ -o segmentedReduce.o segmentedReduce.cu
nvcc  -std=c++11 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70  -o segmentedReduce segmentedReduce.o -L/cm/extra/apps/CUDA.linux86-64/9.1.85_387.26/lib64

./segmentedReduce
...
Derive temp_storage_bytes...
no error
temp_storage_bytes = 1
no error
segmentedReduce output d_out is Correct!

Change Line 34,
#def NUM 5
from 5 to any number <=4. The code will run fine regardless compiling with or without -g -G.

segmentedReduce.cu.txt

Forgot to report the cuda version is 9.1.85_387.26, and cub 1.8.0.

Looking into it

Okay, the issue here is that compiling with -G (device debugging support) can require the runtime to allocate a lot of device storage behind the scenes when launching a kernel.  In this case, because you have already allocated so much memory (“workspace “ + “additional” allocations), you in fact are getting a legitimate out-of-memory exception when running the CUB kernel.  But it’s not CUB’s fault, nor is it a bug.  Your simplest remedies are either:
(a)	not compile with -G, or
(b)	use less memory
To illustrate, if you decrease the amount of extra memory (say to “free -<PHONE_NUMBER>” bytes), everything runs fine.
So…. Not a bug.


combo-box

Please use the form below, leaving the prefilled data to help us. Thank you.
Page link: combo-box
Document link: combobox.md
My Issue/Suggestion
I spotted a typo: the "L" in the word panel on the title and description is missing.

Thank you, Mark, for the explanation. I just learned something new :)


Update Client.php to replace depracated methods for newer PHP version

using iconv_set_encoding is deprecated in PHP>50600. The proposed change checks the PHP version and uses the new method instead when appropriate.

Why not version_compare() for this? http://php.net/manual/en/function.version-compare.php
Example:
if(version_compare(PHP_VERSION, '5.6.0') >= 0) {}


Pre-RTW cleanup: remove unactionable debug log messages

This change prepares for the RTW (release to world) for MRTK v2.0 by removing debug log warnings that are not actionable by the developer (system or app developer).
This addresses #4277

/azp run stabilization_pr


Feature Request: Control over reference lookup from open editors

I've been using GitHub Copilot and it's been a great help in my development process. However, I've noticed that Copilot uses the context of the currently open file to generate suggestions, even when the conversation is not related to the open file. This sometimes leads to less accurate responses.
I'd like to suggest a feature that would give users more control over this behavior. It would be helpful to have a checkbox or some form of control in the settings to toggle whether Copilot should look up references from open editors. This way, users can choose to limit the context to the conversation when the open file is not relevant.
I believe this feature would enhance the user experience by providing more accurate suggestions based on the user's preference.
Thank you for considering my suggestion.

Thanks for this suggestion! It's a great idea, in fact it's exactly what we are working on this month. Follow https://github.com/microsoft/vscode-copilot-release/issues/675, hope you try it out once we have something working in Insiders.


Check litellm token_counter and get_max_tokens to chunk the review

Use the two variables to break the review across multiple files and them merge them.

merged with #89


Feature #224: Activate tokens automatically

This PR closes #224, by:

Adding:

a blacklistedTokens list to the safe's store
a BLACKLISTED_TOKENS key to the persistence layer (immortal)
all the required actions, selectors, reducers required to properly update blacklisted tokens
a specific action activeTokensByBalance which implements the rules to:

persist the list of activeTokens
add all the tokens that have balance > 0
avoid those tokens that were blakclisted

Editing:

how/when Balances and TokenList components loads (fetches) tokens

now Balances fetches tokens from backend every time it is loaded
TokenList no longer fetches tokens but gets those provided by the Balances component

Note: about the new endpoint described in #116, despite it's listed in the swagger documentation, and it seems to be implemented in the safe-relay-service repo's master branch. I wasn't able to access it. I'm getting a 404 (https://safe-relay.staging.gnosisdev.com/api/v1/safes/0x5bF46392B1f0C3C74Df9EB70DC8a6162C71EeC28/balances/)

Travis automatic deployment:
https://pr300--safereact.review.gnosisdev.com
Storybook book automatic deployment:
https://pr300--safereactstorybook.review.gnosisdev.com

Travis automatic deployment:
https://pr300--safereact.review.gnosisdev.com
Storybook book automatic deployment:
https://pr300--safereactstorybook.review.gnosisdev.com

@lukasschor, I have added a modification on how it calculates the "to be active" tokens based on the balance.
The version you tested took the whole list of available tokens returned by the server /v1/tokens/ and checked its balances using web3.
The last change I added uses the list of tokens with their balances returned by /v1/safes/{address}/balances/
This simplifies a lot the process, does fewer requests and uses the endpoint stated in the issue.

Travis automatic deployment:
https://pr300--safereact.review.gnosisdev.com
Storybook book automatic deployment:
https://pr300--safereactstorybook.review.gnosisdev.com

Travis automatic deployment:
https://pr300--safereact.review.gnosisdev.com


Remarks for interlacing and "curved" lines on a wall

Since questions about interlacing and curved curves on surfaces have arisen frequently, this ticket serves to explain this in more detail in the documentation:
Interlacing
How should I interpret the scan rate and lidar resolution from the manual?
What is the relationship between ROS point cloud publishing rate and scan frequency here?
Explanation of curved curves on surfaces
Although the lidar hits a flat wall, the points appear on a curved line. How can this be?

Done.


@everyone mentions don't highlight correctly.

Quarrel Insider 20.7.5.

@Waddle-Doo Do you expand on this?
Is it a visual issue? do they not highlight at all?

Mhm, they don't highlight at all.


tr_model: Fix typo when reading md3 models

MD3 models do some extra processing so it copies name to filename. We were loading name which was causing crashes when trying to view md3 models.
Fixes https://github.com/Unvanquished/Unvanquished/issues/2371

LGTM

To add more color it seems by using name, we would load the same model for all the LODs (whereas if we use filename, the file won't exist and we would not load any code there), which triggers some mad lod code and causes a crash.

I'm playing this on bunker server right now, with 0.54.0/sync. No crashes anymore.


[Firebase_Database] Fix possible NullPointerException

Description
This PR fixes a possible NullPointerException that happens when the plugin is registered without a valid Activity.
I took the opportunity to add some nullability annotations that the IDE was suggesting, those are in the interfaces that are being implemented in the plugin, so should be safe to use.
Related Issues
https://github.com/FirebaseExtended/flutterfire/issues/19
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[X] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[X] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See [Contributor Guide]).
[X] All existing and new tests are passing.
[X] I updated/added relevant documentation (doc comments with ///).
[X] The analyzer (flutter analyze) does not report any problems on my PR.
[X] I read and followed the [Flutter Style Guide].
[X] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[X] I updated pubspec.yaml with an appropriate new version according to the [pub versioning philosophy].
[X] I updated CHANGELOG.md to add a description of the change.
[X] I signed the [CLA].
[X] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[X] No, this is not a breaking change.

@BugsBunnyBR @collinjackson isn't this PR changes FirebaseDatabase plugin behavior contract
from:
Always emit events on Main thread

to:
Always emit events on thread which on which plugin was instantiated

To be compatible with prievious behaviour this should be changed:
private final Handler handler = new Handler();

to this:
Handler handler = new Handler(Looper.getMainLooper());

?

I manually create&register plugins in my app context :)
Of course I do that on main thread, but still I if would make mistake, than previous implementation would work and current not.


TDE5e Core Rules displays wrong version number in Module Management

Describe the bug
In Module Management, the TDE5 Core Rules shows version 3.4.0 when it's been updated to 3.4.5
To Reproduce
Steps to reproduce the behavior:

Update TDE5 Core Rules module to the latest (3.4.5) version
Create or open a new world using TDE5 Dore Rules
Open Module Management and locate TDE5 Core Rules module in the list

Expected behavior
Version displayed should be 3.4.5 and not 3.4.0
Screenshots

Versions:

Foundry Version 9.269
DSA/TDE version 3.4.5

Foundry, the tde system and the core module are three different things. They are allowed to have different versions


Add dead zone to knobs

This fixes the issue https://github.com/Allen-Synthesis/EuroPi/issues/209 by implementing deadzones for the knobs.
Please see the discussion in the issue for details.
This PR does not change:

Anything about ain. I opened an extra issue for that (https://github.com/Allen-Synthesis/EuroPi/issues/211)

This PR does change:

The deadzones are implemented in the percent() function of the AnalogueReader class.
Also the Knob class now actually makes use of the percent() function in the AnalogueReader class instead of completely reimplementing that part.
Also the _sample_adc() function of AnalogueReader was changed for better performance without changing the results
Tests have been modified to account for the deadzones

Awesome work @redoxcode & @gamecat69! Thanks for taking the time to dig in on this problem. Since this change is modifying current firmware behavior and affects all scripts, I want to make sure we take our time with this change. I've added a few first pass comments. I'll test this change on my EuroPi too when I get a chance.

I was still dubious about the need for this change, so I checked this branch out and updated the knob line in the diagnostic script like so:
oled.text(f"{k2.percent():.2f} {k2.percent(deadzone=0):.2f}", 2, 13, 1)
This allowed me to turn the knob and see what value would be retuned both with (L) and without (R) the change. On my hardware the difference was minimal, but noticeable. It turns out that my hardware already exhibits a noticeable deadzone. This is true on both of my modules, one that used 5% resistors, and one that used 1%.
So I guess that my hardware isn't the target demographic for this change, for whatever reason. I think that code changes look good to me now, so I'm going to approve the PR. I think that before we merge this change, it would be nice to have some confirmation from someone with the root problem that this change improves their experience. We might need to tweak the default deadzone value, or make it configurable.

I think there should be further changes to the tests after adding deadzones to read_position() as well? Or am I wrong here?

I think there should be further changes to the tests after adding deadzones to read_position() as well? Or am I wrong here?

Yeah, good call, should be very similar to the other test additions.

Yeah, that's why I didn't mark that part as resolved yet. Just haven't had the time yet.
I guess the tests don't fail cause the deadzones are to small to have an effect in most cases.

Yeah, that's why I didn't mark that part as resolved yet. Just haven't had the time yet.

No worries, take your time. There's no rush here.

ok, think I included deadzones of 0.01 (default) and 0.0 in all the tests now.
Also according to https://github.com/Allen-Synthesis/EuroPi/issues/209 this fixes the issue for people that had it, while not introducing a noticeable difference for people that don't (from my tests).
Anyway the exact default value could be tuned by future PRs.
So.. ready to merge?

Nice work! I can confirm this PR successfully addresses the noise issue on the knobs using the deadzone approach. However, the noise on ain also mentioned in #209 is still present. See more info on how I tested this below. I suspect the noise on ain is still present because this PR only contains the knob deadzone code (as in the title), but I didn't go any deeper into the code to confirm. Suggest the issue remains open until the ain noise is also addressed.
I used the latest version of all packages and copied over the new europi.py.
I then ran this script to test the knob min and max values and ain min values using the previous and new version of europi.py, see results below.
Previous version of europi.py:
k1 min: 0.000366
k2 max: 0.997284
k2 min: 0.000366
k2 max: 0.997269
ain low: 0.004055
Version of europi.py in this PR:
k1 min: 0.0
k2 max: 1.0
k2 min: 0.0
k2 max: 1.0
ain low: 0.004078

Yes, this PR only addresses the issue with the Knobs. The deadzone code could be activated for ain in an other PR.
However I don't think deadzones are the way to go for ain.
I opened a new issue for the ain here: https://github.com/Allen-Synthesis/EuroPi/issues/211


pgsql: Disable hashjoins to get introducing layers for notifications

	// Create a transaction to disable hash joins as our experience shows that
	// PostgreSQL plans in certain cases a sequential scan and a hash on
	// Layer_diff_FeatureVersion for the condition `ldfv.layer_id >= $2 AND
	// ldfv.modification = 'add'` before realizing a hash inner join with
	// Vulnerability_Affects_FeatureVersion. By disabling explictly hash joins,
	// we force PostgreSQL to perform a bitmap index scan with
	// `ldfv.featureversion_id = fv.id` on Layer_diff_FeatureVersion, followed by
	// a bitmap heap scan on `ldfv.layer_id >= $2 AND ldfv.modification = 'add'`,
	// thus avoiding a sequential scan on the biggest database table and
	// allowing a small nested loop join instead.

This commit reduces the execution time of searchNotificationLayerIntroducingVulnerability with certain parameter sets, that took 4-5min as noticed this morning to milliseconds. Let's experiment and verify it doesn't make the general case worse.

Before
SELECT current_setting('enable_hashjoin');
+-------------------+
| current_setting   |
|-------------------|
| on                |
+-------------------+

+-----------------------------------------------------------------------------------------------------------------------------------------
| QUERY PLAN
|-----------------------------------------------------------------------------------------------------------------------------------------
| Limit  (cost=3584098.82..3589355.86 rows=1000 width=92) (actual time=396725.947..397293.267 rows=1000 loops=1)
|   CTE ldfv
|     ->  Sort  (cost=3583581.36..3584098.38 rows=206808 width=8) (actual time=396725.910..396727.252 rows=1000 loops=1)
|           Sort Key: ldfv_1.id
|           Sort Method: quicksort  Memory: 279kB
|           ->  Hash Join  (cost=3054514.44..3562494.35 rows=206808 width=8) (actual time=323790.647..396721.617 rows=3903 loops=1)
|                 Hash Cond: (vafv.featureversion_id = ldfv_1.featureversion_id)
|                 ->  Nested Loop  (cost=0.72..1989.25 rows=229 width=8) (actual time=4.201..5.873 rows=97 loops=1)
|                       ->  Index Only Scan using vulnerability_affects_feature_vulnerability_id_featureversi_key on vulnerability_affects
|                             Index Cond: (vulnerability_id = 1138617)
|                             Heap Fetches: 97
|                       ->  Index Only Scan using featureversion_pkey on featureversion fv  (cost=0.29..6.41 rows=1 width=4) (actual time=
|                             Index Cond: (id = vafv.featureversion_id)
|                             Heap Fetches: 97
|                 ->  Hash  (cost=1656990.09..1656990.09 rows=80396850 width=12) (actual time=323765.432..323765.432 rows=80336226 loops=1
|                       Buckets: 131072 (originally 131072)  Batches: 65536 (originally 2048)  Memory Usage: 11670kB
|                       ->  Seq Scan on layer_diff_featureversion ldfv_1  (cost=0.00..1656990.09 rows=80396850 width=12) (actual time=0.06
|                             Filter: ((layer_id >= 0) AND (modification = 'add'::modification))
|                             Rows Removed by Filter: 741791
|   ->  Nested Loop  (cost=0.44..1087197.88 rows=206808 width=92) (actual time=396725.944..397291.538 rows=1000 loops=1)
|         ->  CTE Scan on ldfv  (cost=0.00..4136.16 rows=206808 width=4) (actual time=396725.915..396729.219 rows=1000 loops=1)
|         ->  Index Scan using layer_pkey on layer l  (cost=0.44..5.23 rows=1 width=92) (actual time=0.557..0.558 rows=1 loops=1000)
|               Index Cond: (id = ldfv.layer_id)
| Planning time: 8.458 ms
| Execution time: 397296.731 ms
+-----------------------------------------------------------------------------------------------------------------------------------------

After
+-----------------------------------------------------------------------------------------------------------------------------------------
| QUERY PLAN
|-----------------------------------------------------------------------------------------------------------------------------------------
| Limit  (cost=3960906.47..3966163.54 rows=1000 width=92) (actual time=75.168..89.405 rows=1000 loops=1)
|   CTE ldfv
|     ->  Sort  (cost=3960388.98..3960906.03 rows=206819 width=8) (actual time=75.150..76.229 rows=1000 loops=1)
|           Sort Key: ldfv_1.id
|           Sort Method: quicksort  Memory: 279kB
|           ->  Nested Loop  (cost=171.25..3939300.92 rows=206819 width=8) (actual time=0.110..70.779 rows=3903 loops=1)
|                 Join Filter: (vafv.featureversion_id = ldfv_1.featureversion_id)
|                 ->  Nested Loop  (cost=0.72..1989.25 rows=229 width=8) (actual time=0.022..1.085 rows=97 loops=1)
|                       ->  Index Only Scan using vulnerability_affects_feature_vulnerability_id_featureversi_key on vulnerability_affects
|                             Index Cond: (vulnerability_id = 1138617)
|                             Heap Fetches: 97
|                       ->  Index Only Scan using featureversion_pkey on featureversion fv  (cost=0.29..6.41 rows=1 width=4) (actual time=
|                             Index Cond: (id = vafv.featureversion_id)
|                             Heap Fetches: 97
|                 ->  Bitmap Heap Scan on layer_diff_featureversion ldfv_1  (cost=170.53..17008.51 rows=14799 width=12) (actual time=0.562
|                       Recheck Cond: (featureversion_id = fv.id)
|                       Filter: ((layer_id >= 0) AND (modification = 'add'::modification))
|                       Rows Removed by Filter: 5
|                       Heap Blocks: exact=4219
|                       ->  Bitmap Index Scan on layer_diff_featureversion_featureversion_id_idx  (cost=0.00..166.83 rows=14947 width=0) (
|                             Index Cond: (featureversion_id = fv.id)
|   ->  Nested Loop  (cost=0.44..1087263.21 rows=206819 width=92) (actual time=75.165..87.677 rows=1000 loops=1)
|         ->  CTE Scan on ldfv  (cost=0.00..4136.38 rows=206819 width=4) (actual time=75.154..78.190 rows=1000 loops=1)
|         ->  Index Scan using layer_pkey on layer l  (cost=0.44..5.23 rows=1 width=92) (actual time=0.005..0.006 rows=1 loops=1000)
|               Index Cond: (id = ldfv.layer_id)
| Planning time: 0.887 ms
| Execution time: 90.395 ms
+-----------------------------------------------------------------------------------------------------------------------------------------


Bower warning about minified files

When I check this project out and run npm install / bower install, I see the following warning/error from Bower:
bower invalid-meta The "main" field cannot contain minified files
It seems like the fix for this is as simple as updating the following line in bower.json from this:
"main": ["dist/fixtable.min.css", "dist/fixtable.min.js"],
To this:
"main": ["dist/fixtable.css", "dist/fixtable.js"],
I'm planning to submit a pull request to fix this, unless there's a particular reason why this should stay the way it is.

Sure, sounds good. I didn't realize that's what the spec called for 👍


cron is updating Webservicex only

Problem. Apparently cron is updating Webservicex only. I made the change using your files only to find that 24 hours later, the Fixer.io rates were overwritten again with all par 1.0 values from Webservicex. What is the fix for this?

I believe you have to set in the admin which to update on the Cron.

Yes. I did find that to be the case. Thank you for responding.
From: Phillip Jackson<EMAIL_ADDRESS>Sent: Saturday, January 30, 2016 10:25 PM
To: philwinkle/Philwinkle_Fixerio<EMAIL_ADDRESS>Cc: webscot<EMAIL_ADDRESS>Subject: Re: [Philwinkle_Fixerio] cron is updating Webservicex only (#8)
I believe you have to set in the admin which to update on the Cron.
—
Reply to this email directly or view it on GitHub https://github.com/philwinkle/Philwinkle_Fixerio/issues/8#issuecomment-177398408 .  https://github.com/notifications/beacon/AP2Am5rDPRV9L08AIMaUerQyNyPSPrZSks5pfZIWgaJpZM4HGjkB.gif

No virus found in this message.
Checked by AVG - www.avg.com http://www.avg.com
Version: 2015.0.6176 / Virus Database: 4489/11409 - Release Date: 01/15/16
Internal Virus Database is out of date.