[VTA] VTA Compilation Script for Intel FPGA

This PR compiles chisel implement of VTA for Intel FPGA, and its tested with de10-nano board from Terasic.
For the driver that works with the design is available in a separate PR #3394 .

The design contains files with extensions .sdc .qpf .qsf .ipx .qsys, which are not allowed by Jenkins.

Thank you @liangfu; can you complement this PR with a compilation guide for Intel FPGA toolchain? The idea will be to extend this guide: https://docs.tvm.ai/vta/install.html and add a section for Intel FPGAs. Information would include what version of quartus is required etc.

the source for the guide is under docs/vta/install.md

Sure, i could write a guide. To be comparable with PYNQ linux environment, I think it will be a long story with linux kernel compilation, bitbake configurations for custom Angstrom filesystem compilation (many components are missing in the distribution provided by Terasic), environment setup for cross compilation of CMA kernel module and the TVM libraries.

are you building your own linux kernel to get the RPC server to execute on the DE10 nano? If that is too long to describe in the install.md, can you prepare a google doc that describes the steps? It should be reproducible so other people can try the DE10Nano example themselves.

That's because the linux kernel version in the disk image provided by Terasic is not compatible with the latest toolchain provided in Intel Quartus Prime. On the other hand, as requested, I would provide a document that describes sufficient details in setting up the running environment for VTA.

In that case I suggest to specify in the guide to install a previous version of Quartus prime to be compatible with that test platform.

Which one did you use to compile the test design?

Currently, I'm using Intel Quartus Prime 18.0 Lite Edition. Let me first reform the scripts to use tcl script, complement this PR with a related document, and then I would inform you when this is ready to review.

Thank you I'll install it on my machine so I can reproduce the experiments.

@tmoreau89 For now, both compilation script and the related documentation have been improved. I think one can easily generate the FPGA hardware for de10-nano with the written document and the compilation script. Please take a review.

Thank you @liangfu for the good work, I'll be reviewing it later today (I installed Quartus to reproduce the build).

@liangfu one question regarding this implementation: does it use coherent accesses to memory? (e.g. on the Zynq we use the ACP port, and set the AxCACHE bits on the AXI master interface to 1111)

@tmoreau89 Yes, definitely. I have set cacheCode = 15, protCode = 4, userCode = 1 to ensure cache coherency. See the changes that I have made here. Specifically, I have followed the recommended setting in this page. In addition, the recommended setting in this page is valuable, too. To quote its recommendation, we have to

pass the physical address + 0x80000000 to the DMA so the access is done through ACP.

Hey @liangfu ,
Nice find about ACP. I did try it on the Pynq and it works, just tried a simple alu test atm. I was missing the prot and user values. By the way, since this will work for both Pynq and DE10 what do you think about updating the Shell config file this way?
I don't think for now at least that the user must know about prot, cache, and user values. They would want to just specify whether is coherent or not. What do you think?

I agree; coherency should be specified as a boolean from which we derive the AxCACHE and AxPROT values on the AXI bus.

@vegaluisjose Agree, I will merge with your PR.

I just merged @vegaluisjose 's PR; please rebase and merge accordingly

@tmoreau89 I made the merge along with some updates in compliance with the linter. Please take another review.

Hey @liangfu
Nice work, I was checking some of the files and I have a question regarding the IntelShell and vta_adaptor_hw.tcl, I was wondering if Quartus has a way of pattern-matching the AXI bus?. We do that currently for backends relying on Xilinx IPI flows (i.e. pynq), it helps reducing some lines to maintain in tcl (see here).
If there is no pattern-matching option available in Quartus, then maybe we can just instantiate VTAShell directly in the Config. By doing this, we could also remove the need for IntelAXI and IntelShell because you are connecting them explicitly in vta_adaptor_hw.tcl.
Otherwise, if some specific naming is required by Quartus for the IP, then we can leave it as it is.
What do you think?

Hi @vegaluisjose , thanks for checking out.
I don't know whether there is any automatic option for matching the pins in AXI bus. If there is any, I would be very glad to use and reform the tcl script.
The reason why I didn't "instantiate VTAShell directly in the Config" is that Intel's implement of AXI bus is slightly different from Xilinx's counterpart. Specifically, it is required to provide arprot and awprot port in Intel's AXILite, although the ports are not connected to any module internally in VTA. Also, Intel only support burst length of 4-bits instead of the default 8-bit in VTAShell. Therefore, I currently instantiated IntelAXILiteClient and IntelAXIMaster in IntelShell. On the other hand, I was just thinking, since there is an XilinxShell implemented, why not just add another IntelShell.
An alternative option I could have is to create arprot and awprot in the commonly used AXILiteClient and AXILiteMaster, and configure VTAShell to use 4-bit burst length directly for de10-nano.
@tmoreau89 What do you think?

Interesting,
We can configure burst length bits in the Config file and that will automatically configure VTAShell. If you need arprot and awprot on AXILite interfaces then lets do that (you can submit a separate PR). I think it will be fine with Pynq, Ultra96, etc. I could test your changes. If we do it that way we can avoid duplicating code/functionality in both TCL and Chisel. I think the idea behind this wrapper is to make you write less code in TCL, but if it does not give you that benefit then it is not worth it to have it.
This is the reason why we are doing the VTAShell approach for AWS F1, because we are not using IPI flow and we need to explicitly connect things.

@tmoreau89 @vegaluisjose To address the review comments, I have minimized the the auto-generated files, derived IntelShell from VTAShell and updated pin assignment in vta_hw.tcl script. Please take another review.

Thank you for the update @liangfu ! I'll reproduce your experiment scripts first before I go ahead with the review.

@liangfu  Looks like the sof compilation succeeded  - one question I have is: does it require Quartus 18.0 or can we rely on the latest version 18.1?

@tmoreau89 I don't think it requires quartus 18.0, it should be applicable on the latest version, although i have not yet tested with 18.1 yet.

can you test it on latest quartus 18.1? I'm thinking it's best to use the latest whenever possible.

sure

@liangfu quick update, I tried building your design with 18.1  and it worked fine. I think it would be good to update the instructions to reflect that. 18.0 has issues when installing on Ubuntu, and I suspect that many users may be using Ubuntu as linux distro.

also please rebase to master; I believe @vegaluisjose fixed the indentation issue in the Scala code, and this will minimize the non-feature  related lines of code

@tmoreau89 I've performed the version update and merged with the latest upstream/master. Please take another look.

@ktabata would you also take a review on this PR?

In order to run Linux on the DE10, where do we find a image to flash the board's SD card? Can you add those instructions to the install.md file as well @liangfu ? Thanks!

the .qsys and .tcl files still have quite a bit of bulk to them, is there a way to minimize that?

I have already removed unrelated components in the .qsys file, it's still large mostly because the detailed settings that related to the HPS configuration (e.g. DDR settings, HPS clocks etc.) is still there. I can try a bit more to figure out whether it can be generated with the help of qsys_tcl.

I'm thinking that these auto-generated files could be dropped in the 3rd party dir under intel/de10 to separate what is user generated and what is part of DE10 packaging (files autogenerated by Intel)

Did you mean I can put them at 3rdparty/intel/de10-nano or vta/hardware/intel/de10-nano? (I think we should avoid using the term de10 instead of de10-nano, because of the existence of DE10-Pro and DE10-Lite)

In order to run Linux on the DE10, where do we find a image to flash the board's SD card? Can you add those instructions to the install.md file as well?

The Linux driver for DE10-Nano and it's related document would be pushed in a separate PR. (See #3394 ) Of course, I can add a brief description about where to download the Linux image, as the user might need them for now.

I can try a bit more to figure out whether it can be generated with the help of qsys_tcl.
That would be great, otherwise see below

Did you mean I can put them at 3rdparty/intel/de10-nano
That's right - it will keep clean the separation between what's custom code, and what's auto-generated scaffolding code

Of course, I can add a brief description about where to download the Linux image, as the user might need them for now.
Yes, just adding instructions on flashing the SD card of the DE10Nano to try some examples would be great. Keeping the PRs separated for hardware and driver is fine by me.

@tmoreau89 I think I have figured out the solution to generate .qsys file with a simple tcl script. In addition, I have also safely removed hps_io related pins, in order to minimize the design to focus on demonstrating VTA subsystem, and added a brief description on flashing SD card image. Please take another review.

@liangfu, thank you for the great work; I'm glad you were able to minimize the auto-generated code footprint. I've added a few more comments before we merge that PR in. Thanks!

Hi @tmoreau89 Thanks for the catch.  I've addressed the review comments. Ready for next round of review.

Thank you @liangfu, I've made one more comment regarding the pkg_config.py file

@tmoreau89 I've made some updates according to your review comments. Please feel free to let me know if there is anything else I should change in this PR.

Thank you for the contribution @liangfu ; this is a very exciting new direction for VTA/TVM!


Esc not working with a complex remap

I'm using the sample for cmd+opt+shift+ctrl from the Elements repo, mapped to Caps Lock. It works for every key that I've tried so far, except Escape. EventViewer sees normal Esc presses, but nothing if I hold Caps first.
{
    "global": {
        "check_for_updates_on_startup": true,
        "show_in_menu_bar": false,
        "show_profile_name_in_menu_bar": false
    },
    "profiles": [
        {
            "complex_modifications": {
                "rules": [
                    {
                        "manipulators": [
                            {
                                "description": "Change caps_lock to command+control+option+shift.",
                                "from": {
                                    "key_code": "caps_lock",
                                    "modifiers": {
                                        "optional": [
                                            "any"
                                        ]
                                    }
                                },
                                "to": [
                                    {
                                        "key_code": "left_shift",
                                        "modifiers": [
                                            "left_command",
                                            "left_control",
                                            "left_option"
                                        ]
                                    }
                                ],
                                "type": "basic"
                            }
                        ]
                    }
                ]
            },
            "devices": [],
            "fn_function_keys": {
                "f1": "vk_consumer_brightness_down",
                "f10": "mute",
                "f11": "volume_down",
                "f12": "volume_up",
                "f2": "vk_consumer_brightness_up",
                "f3": "vk_mission_control",
                "f4": "vk_launchpad",
                "f5": "vk_consumer_illumination_down",
                "f6": "vk_consumer_illumination_up",
                "f7": "vk_consumer_previous",
                "f8": "vk_consumer_play",
                "f9": "vk_consumer_next"
            },
            "name": "Default profile",
            "selected": true,
            "simple_modifications": {},
            "virtual_hid_keyboard": {
                "caps_lock_delay_milliseconds": 0,
                "keyboard_type": "ansi"
            }
        }
    ]
}

This problem is not an issue of Karabiner-Elements.
macOS 10.12 grabs command-control-option-shift-escape in system and generic apps including EventViewer cannot get the event. (like mission control key, command-tab key)

@tekezo is it also not possible to remap the function keys? for example remapping "f4 + alt"


[New game]: Egg Catcher Game

🎮 Game Request
Game logic and basic description
The Egg Catcher game is a sub-game in the Kirby series, debuting in Kirby's Adventure. The game logic is simple: Kirby falls down into Dedede Arena, where King Dedede will begin throwing eggs and bombs at him. The player's goal is to catch the eggs by pressing any button to open Kirby's mouth and releasing the button to close it. The player must avoid catching bombs, as they will end the game.
Point down the features
Game points
Here are the features of the game:
1.Catch eggs thrown by King Dedede to score bonus points and extra lives
2.Avoid catching bombs, which will end the game
3.The number of extra lives rewarded depends on how many eggs Kirby eats
4.In Kirby's Return to Dream Land Deluxe, the game features split-screen multiplayer competition
5.Players can play as different characters, including Kirby, King Dedede, Meta Knight, and Bandana Waddle Dee, each with their own way of catching eggs
6.If Kirby catches all the eggs, the player will be rewarded three lives
7.If Kirby does not catch any eggs, he will stand in the ring sadly in the result screen
8.Unused screens of an Egg Catcher-style sub-game can be found in Kirby Mass Attack's coding
9.Egg Catcher appears as a challenge in NES Remix 2, where Kirby must eat a certain number of eggs to clear the stage.
Select program in which you are contributing
GSSoC24
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @Damini2004 , You can't have another issue before completing the previous one 😀
you already have the following 1 open issues 👀 ! :
#3510

Hey @Damini2004 ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇

Hey @Damini2004 !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

/assign

Why did you close this issue ??
Kindly open this issue


Db threading

Fixed 2 issues about multithreading db access:

only write db when ordered to (no autoflush)
lock threading while writing db

downside: this makes the project even more vulnerable to #44 . If it fails during database flush, all threads will stop

I had reservations about that because it could make workers put duplicated entries to database. But then I remembered that I've been working on a simple app cache to store all current sightings to limit queries sent to database, and this PR looks promising. I'll get to it as soon as I'm done rewriting worker to use tejado/pgoapi directly.

this definitely helped fix the issue by synchronizing it. However when I was doing my own research it seems that scoped_session are recommended to create a thread safe session, But, when I used scoped_sessions I still got the same errors.
http://docs.sqlalchemy.org/en/latest/orm/contextual.html

@modrzew if you want to avoid such database problems in general you will have to implement more thread locks:

whenever you flush something to the database (so no autoflush and every pokemon write
whenever you read from the databse until you have processed the result. So if you check a group of pokemon being in the database you need to lock from the first check until your results have finished writing to the database.

@Aiyubi you need to rebase. Moreover, does it still fit with the most recent additions to master - especially the application cache? There will be much less SELECTs than before.


[RTM] Problem deploying to IIS

I am trying to deploy my asp.net core RTM 1.0.0 application to an IIS server.
When I browse to it, I get this:

How do I figure out what is wrong?
Here are the steps I took..

I did a dotnet publish to a folder.
Copied the published folder to the IIS server.
Installed the net core 1.0.0 preview 2 SDK (x64) onto the IIS server and restarted the server.
Created a new website in IIS, with appropriate bindings, app pool, and with a root directory pointing at the wwwroot directory of the published folder.

One thing I have noticed is that the web.config produced by:
"postpublish": [
      "dotnet publish-iis --publish-folder %publish:OutputPath% --framework %publish:FullTargetFramework%"
    ]

Does not output the web.config to the wwwroot folder of the publish folder. So I manually copied that web.config into the wwwroot folder myself.
Here is the contents of the web.config file:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
    <staticContent>
      <remove fileExtension=".ts" />
      <mimeMap fileExtension=".ts" mimeType="application/x-typescript" />
    </staticContent>
    <aspNetCore processPath=".\Gluon.WebApplication.exe" arguments="" forwardWindowsAuthToken="false" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" />
  </system.webServer>
</configuration>

Is the  "aspNetCore processPath" relative to the web.config itself? I tried changing the processPath to `processPath="..\Gluon.WebApplication.exe" as the exe is one directory back, but that gave the same error - making me think it's not even getting as far as resolving the exe.
Does the SDK installer install the AspNetCoreModule ?

 I'll give that a go..

Yup. Needed to install that.


Add protected namespaces

Pages in a protected namespace can only be edited by the original author. It is used for user profiles.
Admins are able to edit these pages but it's not recommended.

Protected pages are a bad idea. We're not doing this.


Add docker CI

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Package manager should close when connection is switching

And stop refreshing. Even on local if user closes it it shouldn't be eating CPU cycles.

The second part is more a global issue. There are mutated event listeners (in PM, VE, etc.). Probably those listeners should react on event only when corresponding UI is visible.

If this is too complicated, we may leave it alone and document

With fix for #2808 another issue becomes prominent: async update of the package list causes odd behavior: in remote, library link is plain text but on local it is a clickable link. After switching remote -> local or the other way around and then clicking on various packages the link retains old appearance for some time and then eventually updates as view model is rebuilt.

The last one appears by design since fetching package info takes time. Closing for now.


Upgrading to Helm v3

Rancher Server Setup

Rancher version: 2.8.1
Installation option (Docker install/Helm Chart):

1 Instance on Docker single-host
1 Instance as Helm Chart on EKS

In the Rancher Release notes, it is noted that helm v2 is deprecated and will be removed in Rancher v2.9.
When i go to The About Page on the Rancher Web-Interface, it says that I am using Helm Version v2.16.8-rancher2.
My question is, how do i update the helm version? Is it done automatically with the Rancher Image Version?
Or what would i have to update manually before upgrading to 2.9.
It would be helpful to find information about this in the docs, but i couldn't find it.
Thanks for your help and best regards.
Robin

This may be a dashboard/visual issue more than a practical one. You can find our orgs fork of helm here: https://github.com/rancher/helm/releases
While the rancher images do still have V2 for compatibility, they do also have (and default AFAIK) to use v3 of helm. You can see the version included in 2.9 here: https://github.com/rancher/rancher/blob/da175cee0c1c62e6dbc90577bc548f6c9bd3c492/package/Dockerfile#L90
We pretty frequently update it to new helm versions, and then propagate that to our other components. For instance, our rancher/shell images (and other similar components) get updated to use new helm tags as they are released too.

Thanks. I also noticed a higher version when checking the version via the web CLI.


A bug when testing

Thank you very much for making this code available - it is very inspiring.
I just want to forward the pre-trained network, so I downloaded the test.input and checkpoint, then I ran the code python main.py --config configs/shapenet_eval.yaml SOLVER.ckpt logs/shapenet/shapenet/checkpoints/00300.model.pth, but it raised the following error.
I guess there's something wrong with the dataset. Look forward to your reply. Thank you！
Total number of parameters: 2.182M Logdir: logs/shapenet_eval/test Load the checkpoint: logs/shapenet/shapenet/checkpoints/00300.model.pth The start_epoch is 1   0%|                                                     | 0/1 [00:00<?, ?it/s] Traceback (most recent call last):                                                 File "/home/l/t/ognn-pytorch/main.py", line 117, in <module>     OGNSolver.main()   File "/home/l/anaconda3/envs/ognn/lib/python3.10/site-packages/thsolver/solver.py", line 439, in main     cls.worker(0, FLAGS)   File "/home/l/anaconda3/envs/ognn/lib/python3.10/site-packages/thsolver/solver.py", line 428, in worker     the_solver.run()   File "/home/l/anaconda3/envs/ognn/lib/python3.10/site-packages/thsolver/solver.py", line 403, in run     eval('self.%s()' % self.FLAGS.SOLVER.run)   File "<string>", line 1, in <module>   File "/home/l/anaconda3/envs/ognn/lib/python3.10/site-packages/thsolver/solver.py", line 367, in evaluate     self.eval_epoch(epoch)   File "/home/l/anaconda3/envs/ognn/lib/python3.10/site-packages/thsolver/solver.py", line 230, in eval_epoch     self.eval_step(batch)   File "/home/l/t/ognn-pytorch/main.py", line 67, in eval_step     output = self.model.forward(octree_in, octree_out, update_octree=True)   File "/home/l/t/ognn-pytorch/ognn/ounet.py", line 132, in forward     output = self.octree_decoder(convs, octree_in, octree_out, update_octree)   File "/home/l/t/ognn-pytorch/ognn/ounet.py", line 124, in octree_decoder     octree_out.octree_grow(d + 1)   File "/home/l/t/ognn-pytorch/ognn/octreed.py", line 123, in octree_grow     super().octree_grow(depth, update_neigh)   File "/home/l/anaconda3/envs/ognn/lib/python3.10/site-packages/ocnn/octree/octree.py", line 292, in octree_grow     self.nnum[depth] = nnum IndexError: index 4 is out of bounds for dimension 0 with size 4

Clone the latest ognn code and use the lastest ocnn package (pip install --upgrade ocnn).

Thank you very much! I can now run the code successfully.


Email Stats Show Win and Loss to Same Person

I got the Email from Multiplayer  info Email about my stats and it Said Lost to (playername) won against (same name). this might be caused by the fact you win and lose weather the Enemy placed Flags or not. So i think it would be the Most Easy if the code Senats the Email ignores a player already shown in Loss category, because everything else would effect the flag System or the replay Match while user is online possibility.

I think this will be fixed by fixing #1830, so I'll close in favor of that one. I don't want to make the emails hide the problem.


Docs: missing/incorrect docs make it difficult to use

Examples of incorrectness:

import {Switch} from "@material/mwc-tab-bar"

Examples of not very useful:

<mwc-tab-bar>sentiment_very_satisfied</mwc-tab-bar> should show usage with tabs

Examples of undocumented quirks:

Use of icon property does not work without importing mwc-icon separately.

Examples of missing docs:

Events - digging through typings/code is not particularly feasible. For example, even after looking at the relevant code, it's difficult to find the event name. See https://github.com/material-components/material-components-web-components/blob/master/packages/tab-bar/src/mwc-tab-bar-base.ts#L148 which is defined as a string constant

We'll track the effort to add detailed READMEs to all components at https://github.com/material-components/material-components-web-components/issues/348


UI - Move Acknowledgements to the top of Rules page

Please liaise with a designer before actioning this!
cc: @adamcogan @tiagov8 @timdoldissenssw
(copy of email from Tim)
Hi @bradystroud,
As per a conversation with Adam, SSW Rules display “Last updated by” (the editor) at the top of the page while “Acknowledgements” (the author/s) are at the bottom. A viewer might assume the Rule is written by the person that last updated it, as this is the first name they see.

Move Acknowledgments (including profile picture circles) to the top of the page.

See figures below.

Figure: “Last updated by” shown at top of page.

Figure: “Acknowledgements” displayed at bottom of page.
Cheers,
Tim
AB#62117

Done - acknowledgements have now been moved to the sidebar


Fixed search_by_implication_both search_by_implication_consequent search_by_implication_double

In all these features the problem is the same:

The order of steps is wrong. it need to choose the main box in "#advanced_set" before the others options otherwise the options can not be clicked because they are hidden

There's something not completely clear here for me: swapping the order of the steps makes the tests pass?
Those steps are just ticking the checkboxes, nothing else. I'm a bit confused...


Updating datastore URI template for v1beta3.

Also updating the docs link in the README.
Was #1339 but this PR targets the v1beta3 branch instead.

@tseaver PTAL

LGTM


Add SSSOM mapping pipeline to Makefile

Before merging:

[x] Switch to branch, run sh run.sh make all
[ ] Review diff (especially mappings), note oddities here
[ ] Discard all changes (don't commit anything) and merge PR

@matentzn to clarify, I should review the mondo.ssssom.tsv file?
What kind of oddities would I expect to find? I spot checked several rows and things look fine. It looks like some mappings have been deleted or added, but is fine, right?

Yes exactly. By slot check I meant: look at 10-20 rows in the diff, and if they appear more or less correct to you, it's fine.


Add ability to auth based on tls.Config

What
Add ability to preform x509 auth based on tls.Config. Currently CN auth can be done only based on Oracle Wallet file.

seems go-ora is well maintained =D.

Yea, I have same feeling but they made significant progress since last year where I was tried to investigate Oracle integrant.

Do we plan to make a proper change to upstream?

Yea. We can try. Added to my TODO list.


feat: 🎸 Vendor AppPlatform 2022-03-01-preview SDK

…9-05-01-preview sdk folder, don't include py.typed, don't include _metadata.json

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your PR is merged into master branch, a new PR will be created to update src/index.json automatically.
The precondition is to put your code inside this repo and upgrade the version in the PR but do not modify src/index.json.

spring cloud


Add controlplane_network to inventory

This variable needs to be set at the inventory level (rather than using the default value) when public_vlan is enabled.

Successfully tested in the performancelab (rdu3)


Rendering Issue with recent XeLaTex with MacTex 2024

On a recent update I got this rendering error with XeLaTex on MacTex 2024. Any hints how I can solve this?

It's not perfect, but this

is the spacing I have after setting
\renewcommand{\acvSectionTopSkip}{0.0mm}
\renewcommand{\acvSectionContentTopSkip}{0.0mm}

in resume.tex and editing
\newcommand*{\cventry}[5]{%
  \vspace{-2.0mm}
  \setlength\tabcolsep{0pt}
  \setlength{\extrarowheight}{0pt}
  \begin{tabular*}{\textwidth}{@{\extracolsep{\fill}} L{\textwidth - 4.5cm} R{4.5cm}}
    \ifempty{#2#3}
      {\entrypositionstyle{#1} & \entrydatestyle{#4} \\}
      {\entrytitlestyle{#2} & \entrylocationstyle{#3} \\
      \entrypositionstyle{#1} & \entrydatestyle{#4} \\}
    \multicolumn{2}{L{\textwidth}}{\descriptionstyle{#5}}
  \end{tabular*}%
  \vspace{2.5mm} % ADD THIS
}

in russell.cls. I haven't been using much else other than cventry so I don't know what other changes to make, but it's probably similar. Hope this helps.


Change from playing with 6 die to 5

According to everything I'm reading online, like this, the game uses 5 die. Should be a super easy fix.

Reopening...Miss clicked the issue.
According to everything I'm reading online, like this, the game uses 5 die. Should be a super easy fix.


Do a release

Do a release so we can give the zipfile url to the plugin loader as a manual install url. Allows one to skip desktop mode, using git, and all that fuss just to get this plugin.

No, this plugin will soon be obsolete anyway.
Valve added in its beta a settings for modifying resolution for non-steam games.
https://steamcommunity.com/app/1675200/eventcomments/3428948355360837698


Write Hash#to_proc specs

Passes on ruby 2.3.0dev (2015-11-11 trunk 52540) [x86_64-darwin15]

Nice!

Thanks! :smile:


fix: resolve symlinks in system watchers

Try to always use a real path in the watcher map. Thanks to the implementation of realFileSystem.realPath() function, it should work even if the symlinked file is deleted.
Related: #557

:tada: This PR is included in version 7.2.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Media References are removed

Hi I encountered the following issue while using the OpenXML SDK:
Description
I opened the attached file with the OpenXML SDK and saved it using a different name.
If I try to open the saved file Power Point reports that the file is broken and that it can try to repair it.
After repairing the document does not contain the video which was originally in the presentation.
Information

.NET Target:  .NET Core
DocumentFormat.OpenXml Version: 2.13.0-beta2

Repro
Use the following code with the attached file and try to open the saved pptx file.
    var ppt = PresentationDocument.Open("source.pptx", false);
    ppt.SaveAs("saved.pptx");

source.pptx
Observed
The saved file is broken and has to be repaired. The repaired file does not contain the video anymore.
Somehow the OpenXML SDK removes all the media references contained in the pptx file.
Expected
A valid pptx file that is identical to the original document.
I would also be willing to help working on a fix if someone could provide me a starting point why/where the references are removed.
Thanks and kind regards,
Lars

@lklein53 Can you try the latest beta? 2.7.1 is quite old at this point.

@twsouthwick thanks for your quick reply. Sorry messed up the bug report. I already tried using version v2.13.0-beta2

Thanks! So to be clear, it removes them in both?  Trying to understand if it is a regression from an earlier version that may have been working as expected.

Yes, both versions produce a broken pptx without the video in it.

Thanks! At least it's not a regression, but we'll take a look

Hi,
I had some time to look into this today.
In line 1731 of the OpenXmlPartContainer is the following code:
ReferenceRelationshipList.AddLast(newDataPartReference);

I think this has to be replaced by
 child.AddDataPartReferenceRelationship(DataPartReferenceRelationship.Create(child, newDataPart, dataPartReferenceRelationship.RelationshipType, dataPartReferenceRelationship.Id));

At least my example works with this change. If you agree I will create a PR for it. Including a test case

Awesome! Go ahead and make a PR and we can explore it further

https://github.com/OfficeDev/Open-XML-SDK/pull/934 contains the change and a simple regression test

@tomjebo Can you take a look to verify this change makes sense?

@twsouthwick can this issue be closed? Somehow my changes have been merged in June last year already (https://github.com/OfficeDev/Open-XML-SDK/pull/934)


fix(sidenav): switches to translate3d() for content positioning

R: @jelbourn @hansl

@jelbourn Addressed comment


add integration test framework and first integration test

How to categorize this PR?

/area testing
/kind test
/priority critical
What this PR does / why we need it:
Adds a integration test framework that runs tests against a running landscaper intallation.
In addition the first test has been added that tests the first simple tutorial.
Currently the tests can be executed by targeting a cluster with a running landscaper (set env var) and then run make integration-test.
The make test  command has been adjusted so that it only runs unit tests.
Currently the test can only be executed manually.
But I'm currently trying to find the best way to automatically run the test in the pipline
Which issue(s) this PR fixes:
Partially fixes https://github.com/gardener/landscaper/issues/34
Special notes for your reviewer:
I only added one test to not make the PR that big.
Therefore the other tests will be added in a separate PR when this one is merged.
Release note:

A integration test framework has been added that make it possible to run tests against a cluster with a running landscaper.

A integration test has been added that tests the simple nginx-ingress tutorial.

/invite @hendrikKahl @MrBatschner @Diaphteiros

/invite @hendrikKahl @MrBatschner @Diaphteiros

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test

/test


update web copy for vaccination doses

Trello - Update vaccination copy in preparation for Johnson & Johnson vaccines.
The ticket doesn't detail all the necessary updates, so I changed the copy following the same language. Make sure to check:
Home page

Banner copy
Compare: Table header name

Location pages

Header
Compare: Table header name
Chart title
Chart description
Chart labels

The remaining mentions to 1st and 2nd shots are in Learn or the tooltips, and will be updated via the CMS.

 Update Vaccinations for J&J


Always open links in chat in a new tab

Pull Request Checklist

[x] Target branch: Pull requests should target the dev branch.
[x] Description: Briefly describe the changes in this pull request.
[x] Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
[x] Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
[x] Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
[x] Testing: Have you written and run sufficient tests for the changes? -> I did not see any tests
[x] Code Review: Have you self-reviewed your code and addressed any coding standard issues?

Description
Add target="_blank" rel="nofollow" to all links in the chat history window. It is annoying to click a link in a chat and have it unload Open-Webui, so I added this feature. (This)[https://github.com/markedjs/marked/issues/655#issuecomment-383226346] issue in the marked repo describes how to do this properly.
This only applies to links within chat messages. For example, asking a model what is the URL of https://google.com? and clicking the link in your message or the one it produces.

Changelog Entry
Changed

Links in chat open in a new tab instead of the same tab

I believe the issue here is this line:
https://github.com/open-webui/open-webui/blob/90503be2edef1a1f7ce2074286b6316d5cb8868a/src/lib/components/chat/Messages/ResponseMessage.svelte#L72
Which calls this function:
https://github.com/open-webui/open-webui/blob/90503be2edef1a1f7ce2074286b6316d5cb8868a/src/lib/utils/index.ts#L32-L40
This transforms <https://www.google.com> into <https://www.google.com>, and then this is passed in to marked here: https://github.com/open-webui/open-webui/blob/90503be2edef1a1f7ce2074286b6316d5cb8868a/src/lib/components/chat/Messages/ResponseMessage.svelte#L440-L445 which correctly parses it
marked knows how to parse <https://example.com> in the latest version as well as 9.1.6, so if we fix that < replacement, it will work.
I don't know what the impact of adding .replaceAll('>', '>') is to sanitizeResponseContent. Another option is to not replace < with <.

I tried two ways to fix sanitization

Method 1: Replace > with > as well (included in this PR)

Patch

diff --git a/src/lib/utils/index.ts b/src/lib/utils/index.ts
index 03cbc626..2f9b1dc2 100644
--- a/src/lib/utils/index.ts
+++ b/src/lib/utils/index.ts
@@ -36,11 +36,14 @@ export const sanitizeResponseContent = (content: string) => {
 		.replace(/<$/, '')
 		.replaceAll(/<\|[a-z]+\|>/g, ' ')
 		.replaceAll('<', '<')
+		.replaceAll('>', '>')
 		.trim();
 };

 export const revertSanitizedResponseContent = (content: string) => {
-	return content.replaceAll('<', '<');
+	return content
+		.replaceAll('<', '<')
+		.replaceAll('>', '>');
 };

 export const capitalizeFirstLetter = (string) => {

Method 2: Use dompurify after markup parsing (as suggested here)

Patch

commit 9c48a1568d00029bef65b9b653d02fd013f52eef
Author: Austen Adler<EMAIL_ADDRESS>Date:   Thu May 16 16:28:10 2024 -0400

    Use dompurify instead of manual sanitization

diff --git a/package-lock.json b/package-lock.json
index be7ae0dc..80477544 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -12,6 +12,7 @@
 				"async": "^3.2.5",
 				"bits-ui": "^0.19.7",
 				"dayjs": "^1.11.10",
+				"dompurify": "^3.1.3",
 				"eventsource-parser": "^1.1.2",
 				"file-saver": "^2.0.5",
 				"highlight.js": "^11.9.0",
@@ -2730,6 +2731,11 @@
 				"url": "https://github.com/fb55/domhandler?sponsor=1"
 			}
 		},
+		"node_modules/dompurify": {
+			"version": "3.1.3",
+			"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.1.3.tgz",
+			"integrity": "sha512-5sOWYSNPaxz6o2MUPvtyxTTqR4D3L77pr5rUQoWgD5ROQtVIZQgJkXbo1DLlK3vj11YGw5+LnF4SYti4gZmwng=="
+		},
 		"node_modules/domutils": {
 			"version": "3.1.0",
 			"resolved": "https://registry.npmjs.org/domutils/-/domutils-3.1.0.tgz",
diff --git a/package.json b/package.json
index f8096269..ea7de885 100644
--- a/package.json
+++ b/package.json
@@ -49,6 +49,7 @@
 		"async": "^3.2.5",
 		"bits-ui": "^0.19.7",
 		"dayjs": "^1.11.10",
+		"dompurify": "^3.1.3",
 		"eventsource-parser": "^1.1.2",
 		"file-saver": "^2.0.5",
 		"highlight.js": "^11.9.0",
diff --git a/src/lib/components/chat/Messages/ResponseMessage.svelte b/src/lib/components/chat/Messages/ResponseMessage.svelte
index 1ad252f6..b067aac1 100644
--- a/src/lib/components/chat/Messages/ResponseMessage.svelte
+++ b/src/lib/components/chat/Messages/ResponseMessage.svelte
@@ -1,5 +1,6 @@
 <script lang="ts">
 	import { toast } from 'svelte-sonner';
+	import DOMPurify from 'dompurify';
 	import dayjs from 'dayjs';
 	import { marked } from 'marked';
 	import tippy from 'tippy.js';
@@ -441,15 +442,15 @@
 									{#if token.type === 'code'}
 										<CodeBlock
 											lang={token.lang}
-											code={revertSanitizedResponseContent(token.text)}
+											code={token.text}
 										/>
 									{:else}
-										{@html marked.parse(token.raw, {
+										{@html DOMPurify.sanitize(marked.parse(token.raw, {
 											...defaults,
 											gfm: true,
 											breaks: true,
 											renderer
-										})}
+										}))}
 									{/if}
 								{/each}
 							{/if}
diff --git a/src/lib/utils/index.ts b/src/lib/utils/index.ts
index 03cbc626..20c68d84 100644
--- a/src/lib/utils/index.ts
+++ b/src/lib/utils/index.ts
@@ -35,14 +35,9 @@ export const sanitizeResponseContent = (content: string) => {
 		.replace(/<\|[a-z]+\|$/, '')
 		.replace(/<$/, '')
 		.replaceAll(/<\|[a-z]+\|>/g, ' ')
-		.replaceAll('<', '<')
 		.trim();
 };

-export const revertSanitizedResponseContent = (content: string) => {
-	return content.replaceAll('<', '<');
-};
-
 export const capitalizeFirstLetter = (string) => {
 	return string.charAt(0).toUpperCase() + string.slice(1);
 };

Which one would you prefer? I have already tested both and they're about the same. I think dompurify is better long-term, but it's a larger change.

1st option would be preferable for now!

Okay, I pushed patch 1

Couldn't check this earlier. The change seems to be working thank you 🙏

Thanks everyone!


Handle errors for Creating an account with Seed phrase and enabling Seed phrase recovery (#911)

@kcole16 @vgrichina
Implemented flow:

Creating account:

CH is run before addAccessKey, so we will be able to recover the account even if addAccessKey fails
if an error appears while adding a key for Seed phrase
current account state is cleared
user is redirected to /recover-seed-phrase and he is able to recover the account with the same seed phrase
additionally if user was creating a second account (he is already logged in), after clear state for created account, we are selecting the previous account so the user will be still logged in

Enabling Seed phrase as second recovery option:

if an error appears while adding a key for Seed phrase
user is redirected to /profile where he is able to try setup recovery once again
additional global alert appears with information what did actually happens.

@kcole16 please check error messages text.

@marcinbodnar is this ok to be merged? Sorry somehow missed before that you already updated it.

@vgrichina np let me resolve conflicts, and it should be ready to go

@vgrichina ready


The second pair image will fail, do we need clean some cache?

i install the e4s, and run the first pair images(eg. swap-a-to-b.png), it runs well
than i run the secod pait c to d, it fail
i try to swap-b-to-a.png, it will fail too
try to run swap-a-to-b.png(step 1) again, it still runs good

i remove the .gitignore, do not fond some cache, do we need remove some cache for the next pair images?
Thanks

I find that,

the D_recon.png and T_recon.png are ok for a to b
while they are wrong when b to a

@e4s2022 did you have some advice?
thank you!

may is not front view


Mention the 3.0 wiki page in the release announcement emails

24 hours has passed since 'approval: done' was set, but this PR has failing CI tests.  Once the tests pass it will get moved to 'approval: ready to merge' automatically, alternatively please review and set the label manually.

Pushed. Thanks!


NIFI-10715: improve tooltips in parameter providers

Summary
NIFI-10715
Improve adding and removing tooltips in parameter providers and clean up some code.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[ ] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 8
[ ] JDK 11
[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Reviewing now...


SPTextUtil.harSPINRDF() to allow non-SPIN properties on Commands

Fix #2: SPTextUtil.harSPINRDF() to allow non-SPIN properties on Commands
This method should probably move to Command.hasSPINRDF() eventually

Hmm should have made a separate branch I guess...


GZipping files

The documented method for GZipping files breaks my TS compiler and doesn't appear to work when looking at the code. Has this been replaced with toGzip()? I may PR with a documentation update in a few days, but i'm still getting to grips with the package.

Right, i appear to have misread the documentation, there is a gzip option on the createSitemapIndex documentation, which works perfectly, the point still stands about the lack of documentation, happy to create a PR unless i've missed anything?

PR welcome. I've just done a docs overhaul so I probably missed it.

Also feel free to drop any suggestions for improvement in here.

@elmarti I've added some docs but still feel free to open a PR for improvements.


Validing correctness

run_validation.py

Correct as of #19, need to write script to validate on mimic


Kafka topic with many partitions (on Yarn)

Hi, I think I've found an issue reading from Kafka when there are too many partitions, specifically when the number of partitions is greater than spark.port.maxRetries (default 16).
Note that I'm using a CDH Spark cluster in yarn-client mode, so I'm not sure this also repros with MemSQL Spark distro.
In summary, when a container starts it tries to bind to a port starting from an initial port, and fails after spark.port.maxRetries. If the Kafka topic has n partitions, then at least n containers are created, and if n > spark.port.maxRetries the available ports are saturated and eventually containers will fail to start.
I think in these cases it would be better to shut down the pipeline (like a fatal error), such that other pipelines can run without issues.
What happens now is that the exceptions are thrown but "ignored" and data is loaded normally. However I'm not sure if all data is loaded in all cases, as creating a test is kind of laborious... but I'd expect that if you have data in all partitions, some would fail.
Interface logs:
16/04/25 16:24:38 ERROR Lost executor 9 on <hostname.redacted>: Container marked as failed: container_1461362160503_0004_01_000010 on host: <hostname.redacted>. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1461362160503_0004_01_000010
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
	at org.apache.hadoop.util.Shell.run(Shell.java:478)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

16/04/25 16:24:38 WARN Container marked as failed: container_1461362160503_0004_01_000010 on host: <hostname.redacted>. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1461362160503_0004_01_000010
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
	at org.apache.hadoop.util.Shell.run(Shell.java:478)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

16/04/25 16:24:38 WARN Lost task 7.0 in stage 0.0 (TID 12, <hostname.redacted>): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container marked as failed: container_1461362160503_0004_01_000010 on host: <hostname.redacted>. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1461362160503_0004_01_000010
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
	at org.apache.hadoop.util.Shell.run(Shell.java:478)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

16/04/25 16:24:38 ERROR Listener SQLListener threw an exception
java.lang.NullPointerException
	at org.apache.spark.sql.execution.ui.SQLListener.onTaskEnd(SQLListener.scala:167)
	at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42)
	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55)
	at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:80)
	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:64)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1181)
	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
16/04/25 16:24:39 ERROR Lost executor 27 on <hostname.redacted>: Container marked as failed: container_1461362160503_0004_01_000030 on host: <hostname.redacted>. Exit status: 1. Diagnostics: Exception from container-launch.

Container logs from:
http://...:8042/node/containerlogs/container_1461362160503_0004_01_000010/stderr/stderr/?start=0
16/04/25 16:24:30 INFO executor.CoarseGrainedExecutorBackend: Started daemon with process name<EMAIL_ADDRESS>16/04/25 16:24:30 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
16/04/25 16:24:32 INFO spark.SecurityManager: Changing view acls to: yarn,emanuele
16/04/25 16:24:32 INFO spark.SecurityManager: Changing modify acls to: yarn,emanuele
16/04/25 16:24:32 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, emanuele); users with modify permissions: Set(yarn, emanuele)
16/04/25 16:24:33 INFO spark.SecurityManager: Changing view acls to: yarn,emanuele
16/04/25 16:24:33 INFO spark.SecurityManager: Changing modify acls to: yarn,emanuele
16/04/25 16:24:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, emanuele); users with modify permissions: Set(yarn, emanuele)
16/04/25 16:24:34 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/04/25 16:24:34 INFO Remoting: Starting remoting
16/04/25 16:24:34 WARN util.Utils: Service 'sparkExecutorActorSystem' could not bind on port 10014. Attempting port 10015.
16/04/25 16:24:34 ERROR Remoting: Remoting error: [Startup failed] [
akka.remote.RemoteTransportException: Startup failed
	at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)
	at akka.remote.Remoting.start(Remoting.scala:194)
	at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
	at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
	at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
	at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
	at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
	at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121)
	at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53)
	at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:52)
	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1989)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1980)
	at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:55)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:266)
	at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:217)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:183)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:148)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:250)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: <hostname.redacted>/<IP_ADDRESS>:10014
	at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
	at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:391)
	at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:388)
	at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
	at scala.util.Try$.apply(Try.scala:161)
	at scala.util.Success.map(Try.scala:206)
	at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
	at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
	at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
	at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
	at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
	at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
	at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
	at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
]

[...]

16/04/25 16:24:35 INFO Remoting: Starting remoting
16/04/25 16:24:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@<hostname.redacted>:10025]
16/04/25 16:24:35 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutorActorSystem@<hostname.redacted>:10025]
16/04/25 16:24:35 INFO util.Utils: Successfully started service 'sparkExecutorActorSystem' on port 10025.
16/04/25 16:24:36 INFO storage.DiskBlockManager: Created local directory at /mnt/jbod_1/yarn/nm/usercache/emanuele/appcache/application_1461362160503_0004/blockmgr-20ef5072-5398-4e44-b38d-710038d26c7f
16/04/25 16:24:36 INFO storage.DiskBlockManager: Created local directory at /mnt/jbod_2/yarn/nm/usercache/emanuele/appcache/application_1461362160503_0004/blockmgr-f7e45ba1-1e24-49fb-a475-74057513f119
16/04/25 16:24:36 INFO storage.DiskBlockManager: Created local directory at /mnt/jbod_3/yarn/nm/usercache/emanuele/appcache/application_1461362160503_0004/blockmgr-d8bdba6b-4296-49fd-ada2-b66b9a379b1c
16/04/25 16:24:36 INFO storage.DiskBlockManager: Created local directory at /mnt/jbod_4/yarn/nm/usercache/emanuele/appcache/application_1461362160503_0004/blockmgr-947725bd-a63a-4f23-ae19-05a0a1722b40
16/04/25 16:24:36 INFO storage.DiskBlockManager: Created local directory at /mnt/jbod_5/yarn/nm/usercache/emanuele/appcache/application_1461362160503_0004/blockmgr-c8d2b2cf-de63-42f5-a2e0-7390066f65ba
16/04/25 16:24:36 INFO storage.DiskBlockManager: Created local directory at /mnt/jbod_6/yarn/nm/usercache/emanuele/appcache/application_1461362160503_0004/blockmgr-3bffbf82-2b57-4f5f-9e60-2a9bfec07120
16/04/25 16:24:36 INFO storage.MemoryStore: MemoryStore started with capacity 530.3 MB
16/04/25 16:24:36 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@<IP_ADDRESS>:10013
16/04/25 16:24:36 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
16/04/25 16:24:36 INFO executor.Executor: Starting executor ID 9 on host <hostname.redacted>
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10011. Attempting port 10012.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10012. Attempting port 10013.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10013. Attempting port 10014.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10014. Attempting port 10015.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10015. Attempting port 10016.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10016. Attempting port 10017.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10017. Attempting port 10018.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10018. Attempting port 10019.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10019. Attempting port 10020.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10020. Attempting port 10021.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10021. Attempting port 10022.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10022. Attempting port 10023.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10023. Attempting port 10024.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10024. Attempting port 10025.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10025. Attempting port 10026.
16/04/25 16:24:36 WARN util.Utils: Service 'org.apache.spark.network.netty.NettyBlockTransferService' could not bind on port 10026. Attempting port 10027.
16/04/25 16:24:36 ERROR netty.Inbox: Ignoring error
java.net.BindException: Address already in use: Service 'org.apache.spark.network.netty.NettyBlockTransferService' failed after 16 retries!
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
	at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
	at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
	at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
	at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
	at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)
16/04/25 16:24:36 ERROR executor.CoarseGrainedExecutorBackend: Received LaunchTask command but executor was null
16/04/25 16:24:36 INFO storage.DiskBlockManager: Shutdown hook called
16/04/25 16:24:36 INFO util.ShutdownHookManager: Shutdown hook called

hmm it looks like those container exits are happening outside of the interface stack so we might have to detect that we're running in yarn to prevent this.
i also remember seeing a config option to tell spark to start trying ports from a different initial value - i don't know if this is still around and it's not the most robust but at least you'll be able to load data. thoughts?

Yes, this is kind of weird, it seems pretty hard to detect the failure at app level.
For my case specifically, I can either increase this maxRetries or look into this randomness (although every default value related to port allocation seems to be random, so I guess this might be a specific issue of Kafka integration... have to look into that).
What I'm kind of scared is how many similar cases Spark/Yarn are hiding. A nice to have could be a sort of "transaction" where either all containers start, or all are killed, but I have no idea if this is supported :/

for what its worth this doesn't seem like an issue with kafka - i think yarn containers consume some ports and so starting new executors in an existing yarn cluster decreases the number of available ports for spark and its various web servers to bind to. maybe there is something that can be done in yarn to get around this?

I'll try to look into more details.
I was saying Kafka in the sense that maybe the way containers are requested for the DStream computation forces this incremental try... but I have no idea, maybe it's a wider issue and I just noticed it in this case.

if what i said is correct, you should be able to repro this issue by creating a vanilla rdd with a large number of partitions (how many partitions are in your kafka topic?) and when it gets processed the same number of executors should be created.

Closing for now, this is also caused by dynamicAllocation, which I have finally turned off :)


Show both a message and a menu?

I wonder if it would be possible to build a bottomsheet with both the message (would be displayed at the top) and the menu (would be displayed below the message). I've been asked to have both a title and a short description message within the bottomsheet.

With the current setup, no. You can achieve this by setting a custom view. But you would have to manage everything yourself (setting up the adapter, click events, styles, etc...)

I found a dirty way to get this effect, by making a layout in my project called bottom_sheet_layout.xml and changing the layout to accommodate my description message above the menu. I read through your source code, and it might be nice to change it so that if menu AND message are set, use a third layout for the bottom sheet. If I have time, I may PR that some day.


Wrong API token name in documentation

In the README.md it says:
Once you have your token add it to your NEXT_STRAPI_API_TOKEN variable name in the .env file.
The name of the token must have changed since it was written. I believe it is now called: NEXT_PUBLIC_STRAPI_API_TOKEN?

@spaceo Thanks you are correct.  I will make the update.


Unaddressable outgoing packet permanently clogs socket TX queue

Steps:

Send a packet to a device running smoltcp with an IP source address X.
Device: respond to the packet (maybe you're doing UDP echo, as a simple example).
 (I've only had this happen on IPv6 but I suspect it would apply on IPv4+ARP as well.)
Do this a few times to fill up the sender's TX queue on that socket.

Expected:
Socket would keep working -- perhaps the unaddressable packets would be dropped eventually.
Observed:
Socket is now permanently clogged, unaddressable packets are tried forever, no further data can be sent.
Musings:
So, I really I appreciate the fact that smoltcp does not discard packets after some arbitrary timeout, since timeouts are always wrong. I also appreciate that there is not some fixed "packet retry counter" that can be exhausted. I've used libraries that made both of these decisions and I had to fix it.
That being said, this is a case where the stack can be DDoS'd pretty easily, so I've been thinking about how to fix it without introducing arbitrary timeouts or counters. My initial thoughts were:

Allow a policy to be specified on each socket that controls behavior when the TX queue is filled -- essentially a cache eviction policy. For instance, if you're doing an "isochronous"-style time-sensitive data stream, you probably want new packets to simply overwrite the oldest enqueued packet. In other cases one might insist on retrying older packets and drop new ones (current behavior).
Invoke some sort of "strategy" (in the gang-of-four sense, could be a function pointer or implementation of some trait) in situations like unaddressable packets, giving it the opportunity to request that the packet be killed, instead of swallowing the event inside the stack. This would let the application apply an arbitrary timeout/counter if that's appropriate for their use case.

Interesting!
Another issue is: even if we add a timeout, UdpSocket only tries to send the packet at the front of the queue. If that's unaddressable, it'll clog up the queue for the duration of the timeout even if other packets in the queue can be sent.
I wonder how Linux handles this...

I'm comfortable with the socket only trying to send the front of the queue, because it provides the property that packets sent on a socket emerge (at least, from the network stack) in the order they were sent. They may of course get reordered elsewhere, but that's a nice property to start with, and in systems like mine where reordering won't happen it's a real plus.

even if we add a timeout

I don't think adding a timeout per se is the right solution because, with rare exceptions, the choice of such a timeout is likely to be wrong. I can expand on this if you like. Providing either an enumeration of handling/evictions policies or the pluggable strategy I suggested seem like better options. (It's possible you were using "add a timeout" as shorthand for the entire space of options, in which case, ignore me -- I just want to make sure a timeout doesn't appear here as a reflex.)

I wonder how Linux handles this...

I don't know off the top of my head, and my initial searches are not bearing fruit, but I would not be surprised if the the answer were "a timeout." ;-)

it provides the property that packets sent on a socket emerge (at least, from the network stack) in the order they were sent

it shouldn't matter if packets for different dest IPs get reordered though, right? as long as all packets for the same IP are not reordered between themselves.

Providing either an enumeration of handling/evictions policies or the pluggable strategy I suggested seem like better options. (It's possible you were using "add a timeout" as shorthand for the entire space of options, in which case, ignore me -- I just want to make sure a timeout doesn't appear here as a reflex.)

Yeah I worded it poorly, I think the single FIFO queue is an issue for many such strategies. For example if we did "if queue full, drop oldest packet": say the queue size is 32, the evil client fills it with 32 poison packets, then a legit client wants a request server. First legit response will overwrite one poison, but there's still 31 others in the queue in front of it, so it won't get sent. The legit client would have to do 32 retries to unclog the queue.

I would not be surprised if the the answer were "a timeout." ;-)

:smiling_face_with_tear:

it shouldn't matter if packets for different dest IPs get reordered though, right? as long as all packets for the same IP are not reordered between themselves.

...yeah, I feel like that might be okay? Implementing that sounds a lot more expensive than always sending front of queue, but maybe there's a trick I haven't thought of.

The legit client would have to do 32 retries to unclog the queue.

You're right, that would still be unpleasant! Now that you mention it, it seems like it would be nice if unaddressable packets could be bumped to the back of the queue. That way we could have many concurrent NDP enquiries in flight, and a "cache eviction" mechanism could allow new traffic to escape if permitted.

(After writing that, it occurs to me that there would be subtleties in determining when to return "no activity" from poll if unaddressable packets get bumped to the back of the queue, since rate-limiting the front packet and checking its addressability would no longer suffice; you'd kinda need to keep track of how many such packets you evaluated. Hmmmmm.)

Yeah, the SocketMeta NeighborState "silencing" assumes each socket only talks to one peer, which is not true for non-tcp sockets. I also ran into this for the WIP DnsSocket.

Is that assumption even valid for TCP? If some other node sends an unexpected packet to the port, doesn't the socket need to establish addressability to send RST? Or does that avoid the full NDP/addressability lookup because it's inside the stack and can simply reuse the original MAC?

RSTs are sent by Interface when no socket matches. If it's unadressable it gets dropped, but that shouldn't happen because the incoming packet would've filled the neighbor cache.

This is biting us quite regularly now -- I'm surprised more users don't appear to be hitting it. It's quite simple to produce if there's ever a case where a machine that was once addressable leaves the network, due to partition or disconnection.
(We are, for what it's worth, still on v0.8, but I don't see anything in the changelog that suggests this has been fixed in v0.9.)
I've tried out a fix in the form of evicting the oldest packet from the outgoing queue on a socket in the event that the queue is discovered full -- basically, switching from a "drop newest" policy to a "drop oldest" policy -- but the naive version of this is insufficient because the socket meta neighbor_state remains permanently stuck in Waiting for the original, unaddressable, address. Because of the way the code is factored, it is currently difficult for the layer that manages the neighbor_state to discover that an eviction has occurred in the tx_buffer in a different type.
Patching the Meta::egress_permitted routine to set the state to Active on timeout increases the chances of noticing that the unaddressable packet has been removed and future neighbor cache checks should use a different address, but if unaddressable packets are being generated regularly you can wind up with another one at the head of the queue.
For instance: given a task that sends UDP broadcasts every 500 ms, altering it to send an unaddressable UDP packet just before each broadcast reproduces the bug reliably. Altering smoltcp to attempt to make space in the UDP socket tx_buffer before returning QueueFull will occasionally leave the queue with a valid packet at the head, but only if the queue depth is odd since packets are being sent in pairs. This seems like it introduces a new error-prone case, and I don't love it.
So, it's starting to feel like the right thing to do is amend NeighborState with a field for tracking how long we've been waiting on the particular neighbor. If this exceeds some limit, we must assume we have a poison packet at the head of the queue, and take recovery action, which would consist of (at minimum)

dequeueing the head packet and discarding it
switching the neighbor_state back to Active.

(I'd love to increment a counter when this happens, but smoltcp's trace infrastructure is heavily text-message oriented rather than event-oriented, so that'd be hard to integrate.)
An extended solution might add an "unreachable" entry type to the neighbor cache, such that when we have to evict a packet due to neighbor resolution timeout, we could cache that fact and turn further transmissions to that address into prompt errors for a certain time period. This would reduce the cost of software that repeatedly attempts to send to an unaddressable peer by amortizing the wait-for-neighbor delay over the number of sends that are denied promptly. Our software doesn't behave this way, though, and I'm assuming other users' doesn't either since apparently nobody's sending to unaddressable peers but me.
The thing I'm proposing is basically a timeout, which is a thing I was trying to avoid. However, in this case, I think there's a good justification for treating a neighbor that is not promptly discoverable as undiscoverable -- because neighbor discovery is a local network operation, it's more predictable than a lot of network operations, and waiting e.g. 3x the current 3-second period should cover most cases.

Agreed. Packets getting stuck behind other packets is a result of smoltcp's admirably simple handling of queues, where only the head element is ever acted upon. Bumping the head element to the rear of the queue if it needs neighbor discovery is one option, I suppose, and that doesn't compromise the simplicity too much. In particular it's still a FIFO, just an occasionally-cycled FIFO. As you noted, if all packets in the queue are discovered to not be ready, the socket would silence itself
If the neighbor cache is extended with a PENDING entry type, adding an UNREACHABLE entry type would reduce the cost of repeated sends to an unaddressable peer; based on Linux's behavior I believe they do this with some timeout.
The potential mass-unsilencing behavior doesn't bother me, personally; we already wind up polling every socket for architectural reasons. (Architectural reasons that could likely be fixed, but currently they're lower priority than the thing where smoltcp sockets clog up and cannot be used again until a reset.)

Bumping packets to the back of the queue would mean packets destined for the same peer could get reordered, though.

Maintaining the order of packets to a single destination would be a nice property. Random-access dequeueing with the current ring data structures is possible -- thankfully they are not fancy concurrent Lamport queues or something -- but would likely require the payloads to be copied to remain contiguous, lest it create fragmentation that blocks further packets.
Memcpy is faster than "stuck forever" so this sounds acceptable to me.

Also, I wanted to say that I greatly appreciate your thoughtful and detailed discussion of this issue. :-)

Update from our side:
We've implemented a rather hacked-up watchdog on top of smoltcp to catch and unclog this situation. The implementation is a little scattered around our network stack, but the part that takes action is here. Essentially what we do is:

Keep track of the fact that we have observed a "queue full" error when attempting to send an outgoing packet on a particular socket.
Record the time at which that error occurred.
If the situation does not resolve itself within [arbitrary time interval], do this:

                // Reset the queue by closing + reopening it.  This will lose
                // packets in the RX queue as well; they're collateral damage
                // because `smoltcp` doesn't expose a way to flush just the TX
                // side.
                let s = self.get_socket_mut(socket_index).unwrap_lite();
                let e = s.endpoint();
                s.close();
                s.bind(e).unwrap_lite();

It's a little gross, yes, and results in loss of any other outgoing packets, plus any incoming packets on the same socket. However, in our application, we strongly emphasize application-layer end-to-end controls for things like redelivery/retransmission, and expect some amount of packet loss at L3 and below (we're all UDP, in fact). So for us, it is relatively easy to recover from losing a few packets in either direction -- certainly easier to recover from than a socket being stuck forever!
Anyway, I post this in case it's a useful workaround for anyone else struggling with this issue.

I can confirm that this issue is present in UDP sockets with ARP requests as well.
For us it's undesirable to miss/lose incoming packets (RX) even if we set the buffer to single packet.
For re-transmission we also plan to add this in our protocol on top of the embassy-net implementation but seems cumbersome to handle the smoltcp re-transmission on our protocol level.
Would it be acceptable to expose the queue or a way to build a custom clean-up logic which can be either timeout or counter based or just drop the packet if destination is unaddressable?
I'm just getting into the smoltcp stack code-base and any guidance is welcome.

I still want to bump this issue. It's a major issue if you expect some of your nodes to go offline (like IoT devices and such).
Would a setting for EvictionPolicy be a good approach to solve this issue and leave it to the user to decide what to do?
E.g.

Eviction policy for timeout
Eviction policy for directly dropping any ARP requests instead of leaving them in the buffer
Eviction Policy for doing nothing? i.e. the current behaviour


Adds the ?shop command

It adds especially a JSON file listing all the items, as of the 1.9.1 update.
The list is lazily loaded when the command is actionned for the first time.
Each time the command is used, the bot choices items randomly in the list.
Since the actual PRNG used in the game is not known, this command currently uses the xorShift32 algorithm as a placeholder.
The xorShift32 algorithm is initialized here with Knuth's multiplicative method.
An alternate algorithm, xorShift128, initialized with a splitMix64, is also provided for testing.
An upcoming commit in this PR might replace the current implementation of the PRNG with the one used in the game.

For now, the ?shop command will only land on moderators channels for testing purposes.


[minor] Add method for setting a "RefreshAddins" registry key.

Thank you for your pull request!
Please add '[major]', '[minor]', or [patch] to the title to indicate the impact the change has on the code.  Please also provide the following information.

Change Description:
For the json manifest scenario, removing the cache modification call that were added as a workaround and replace it with setting a registry key that will signal to the host app that a refresh of add-ins is needed.  This will work for both sideloading and unacquiring.

Do these changes impact command syntax of any of the packages? (e.g., add/remove command, add/remove a command parameter, or update required parameters)
No.

Do these changes impact documentation? (e.g., a tutorial on https://learn.microsoft.com/office/dev/add-ins/overview/office-add-ins)
No.

If you answered yes to any of these please do the following:
> Include 'Rick-Kirkham' in the review
> Make sure the README file is correct
Validation/testing performed:
Ran automated tests.  Create local package that was installed with a test project and used to verify registry was set.  This was test in combination with the host change to read the registry value and clear it.

Can I get a review?


Error: Cannot read property 'token' of undefined

🐛 Error: Cannot read property 'token' of undefined
tips occurred when I use advanced options menu or upload menu. I have tried to reinstall or set a git  but not work.

🌴 Visual Studio Code Version :
🌴 Code Settings Sync Version : [3.2.0]
🌴 Standard or Insiders : [standard]
🌴 OSS or Portable : [ here ]
🌴 Operating System : [windows 10 1803 ]
🌴 Occurs On: [Upload /  Help Configuring ]
🌴 Proxy Enabled: [  No ]
🌴 Gist Id: [ none ]
📺 Console Error Log

Try reset the Settings Sync settings

Try reset the Settings Sync settings

I deleted all the Sync settings in user settings , it does not work too.
thank you

by the way , vscode is in portable mode

what is the path for data folder and extension folder for portable code

what is the path for

Code executable
User directory
Extensions directory

1.D:\VSCode
2.D:\VSCode\data\user-data
3.D:\VSCode\data\extensions
like this.

what is the path for

Code executable
User directory
Extensions directory

Dear shanalikhan,
There is a bug about this issue.
diff --git a/src/environmentPath.ts b/src/environmentPath.ts
index 2230aab..b373ebd 100644
--- a/src/environmentPath.ts
+++ b/src/environmentPath.ts
@@ -79,7 +79,7 @@ export class Environment {
}`;
 process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

this.PATH = process.env.APPDATA;

this.PATH = this.isPortable ? process.env.VSCODE_PORTABLE : process.env.APPDATA;
this.OsType = OsType.Windows;
if (!this.PATH && !this.isPortable) {

I resolve it after rebuild.

@CyrilTaylor Thanks, i have pushed the changes you have asked, window os is not handled properly.
Will release new version soon.
You can look into the commit here
https://github.com/shanalikhan/code-settings-sync/commit/7f1aacec381a2a57415e0b00fef5067a272155c3#diff-09ef59e49c66636eacf529be73517d36R108

v3.2.1 released.
Please verify.


WIP: Improve runtime/syntax/sshconfig.vim (1)

Problem: Syntax file sshconfig.vim is outdated.
Solution: Update sshconfig.vim partially (with no intention to be exhaustive). (Samy Mahmoudi)
Note: I started this work long ago, after creating pull request #3258. Recently, Bram Moolenaar made me notice Kevin Locke's pull request #5753, so I decided to push my changes even if they will not be reviewed (maintainer timeout) to avoid such duplicate efforts.
WIP: Some changes conflict with Kevin's first commit (Kevin adds some MACs to sshdconfigMAC while I add the same MACs to sshconfigDeprecated to mark them as deprecated) but I am sure we will sort this out quickly. Besides, I also came to the conclusion that some MACs were inserted and have been kept in the file whereas they were erroneous (written as mac-* instead of hmac-*) from the beginning ;-)

I wonder if @dffischer is still active.  Last message was more than a year ago, mentioning "will care for it shortly"...

@brammool
For your information, I asked Dominik to take over maintainership when I sent him these patches this morning.

@samy-mahmoudi can you please rebase?

closing for now.


Latest nswag npm package (9.5) fails on Linux (ubuntu/mint)

Steps to reproduce:
Install .net core
Install nswag using sudo install -g nswag
Run nswag from command line
Expected result
nswag cli runs
Actual result
/usr/bin.env: 'node/r': No such file or directory
The problem seems to be that the nswag.js file is using windows line endings which are not interpreted correctly on Linux.
I have confirmed that changing line endings to support Linux fixes the problem.

Should be fixed with v9.5.1

@rsuter: Could we avoid setting everything to binary? Instead we could specify text=auto to automatically convert text files to LF? i.e. use a combination of the C#, Visual Studio and Web templates from here https://github.com/alexkaratarakis/gitattributes
Currently it's hard to make pull requests because the commit line endings will depend on each user's git autocrlf setting. I didn't realize why my diffs were unusual until I saw the recent CR commit.


Add key action info to Android EventHandler events.

Summary
Add event key action for Android EventHandler events, helps to know if the key event is a up (key release, value is 1) or down (key pressed, value is 0) action.
Changelog
[ANDROID] [ADDED] - Add event key action to EventHandler events.
Test Plan
No test added

@aamalric Before I can merge this commit, would you mind signing the CLA at code.facebook.com/cla? It only takes one minute. Thank you!

@cpojer Yes for sure we will soon sign the CLA

This results in double onPress on TV devices when pressing button on a remote. Was this intended? I am guessing to since the doc was not updated: https://facebook.github.io/react-native/docs/building-for-apple-tv ?

@dblachut-adb Hi, yes now when you press one button on TV, you'll have 2 events onPress but with 2 differents events:

one will be for the press (eventKeyAction = ACTION_DOWN = 0)
one will be for the release (eventKeyAction = ACTION_UP = 1).

You'll have to make the check on this event if you need to handle the press only.

IMHO this is not good as I would not normally expect to receive two events for single press in one handler. Correct me if I am wrong but in web environment there are two separated events for it. In non-tv RN onPress is fired one on single user touch and release, but you have an option to listen for touch down and up and others separately.
As this is a breaking change and a bit counterintuitive it would be great to provide separate callbacks or if this is really desired it should be documented.
Also - do you know how is this handled on AppleTV? Didn't this commit broke the compatibility between those two platforms? I have just started to wonder...

On TV you have to use the TVEventHandler wich allows you to listen to events for events not specifically 'onPress' like you said.
Take a look at https://facebook.github.io/react-native/docs/building-for-apple-tv
The TVEventHandler is enabled with a callback that will receive an event, then depending on this event parameters, it's up to you to call your own 'onPress' or 'onRelease' functions...
There's no existing mechanism of onPress/onRelease for TV inputs in RN for now as far as I know

Yes there is and it is specifically described in the link that you have provided. I am talking about Touchable* elements (Buttons) onPress property. Touchables can be focused using directional navigation and then after you press OK (select) on remove you will receive onPress callback.

The Touchable mixin has code added to detect focus changes and use existing methods to style the components properly and initiate the proper actions when the view is selected using the TV remote, so TouchableWithoutFeedback, TouchableHighlight, TouchableOpacity and TouchableNativeFeedback will "just work". In particular:

onFocus will be executed when the touchable view goes into focus
onBlur will be executed when the touchable view goes out of focus
onPress will be executed when the touchable view is actually selected by pressing the "select" button on the TV remote.

You are talking about 'global' event handler, maybe that is where we don;t understand each other. Does your change affect only global event handler?

My changes were originally made to detect a press from a release on TVEventHandler events.
We don't use Touchable on our TV version as we didn't like the way the spatial navigation was handled with a TV remote...
So I must admit that I didn't test the Touchable handlers behavior with my modifications.

I can see that somebody have already filed an issue for it: https://github.com/facebook/react-native/issues/25979

I also have the same problem. Using the mouse works fine, but using a keyboard or box control onPress runs twice

This was a pretty major breaking change for us that we did not anticipate...

This was a pretty major breaking change for us that we did not anticipate...

@mtkopone Are you sure that this change is the cause? As I have mentioned - it is not yet released under any version.

Looking at the code, I think the problem is in this "OR" of print. It checks two button click actions when you press the button and when you release it.
I may be wrong, but I think the problem is because of this. I think I needed to check only 1 of the actions.
https://i.imgur.com/kawDqJ1.png

That was my first assumption as well, but this commit is not yet released to any RN version which means that you would have to:

build react-native from sources
have sources checked out to master branch or have this commit cherry picked

This thread is a year old and I still don't understand why onLongPress, onPress, onPressIn, onPressOut are handled as you would expect by touch but then onLongPress, onPressIn, onPressOut have no effect on a keypress, instead just firing onPress 3 times and expecting you to deal with it?


Crash - Entering Field Offices will crash the District.

Entering a Sellbot Field Office will crash the District you are playing in.
  File "C:\Open-Panda\python\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Open-Panda\python\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "E:\Toontown\Toontown Fantasy\Toontown-Fantasy\toontown\ai\AIStart.py", line 78, in <module>
    run()
  File "E:\Toontown\Toontown Fantasy\Toontown-Fantasy\otp\ai\AIBase.py", line 202, in run
    self.taskMgr.run()
  File "C:\Open-Panda\direct\task\Task.py", line 541, in run
    self.step()
  File "C:\Open-Panda\direct\task\Task.py", line 492, in step
    self.mgr.poll()
  File "C:\Open-Panda\direct\distributed\ConnectionRepository.py", line 612, in readerPollUntilEmpty
    while self.readerPollOnce():
  File "C:\Open-Panda\direct\distributed\ConnectionRepository.py", line 617, in readerPollOnce
    if self.checkDatagram():
  File "E:\Toontown\Toontown Fantasy\Toontown-Fantasy\toontown\cogdominium\DistributedCogdoInteriorAI.py", line 380, in setAvatarJoined
    self.fsm.request('Elevator')
  File "C:\Open-Panda\direct\fsm\ClassicFSM.py", line 281, in request
    self.__transition(aState,
  File "C:\Open-Panda\direct\fsm\ClassicFSM.py", line 230, in __transition
    self.__enter(aState, enterArgList)
  File "C:\Open-Panda\direct\fsm\ClassicFSM.py", line 216, in __enter
    aState.enter(argList)
  File "C:\Open-Panda\direct\fsm\State.py", line 200, in enter
    self.__enterFunc(*argList)
  File "E:\Toontown\Toontown Fantasy\Toontown-Fantasy\toontown\cogdominium\DistributedCogdoInteriorAI.py", line 486, in enterElevator
    self._game = self._createGame()
  File "E:\Toontown\Toontown Fantasy\Toontown-Fantasy\toontown\cogdominium\DistributedCogdoInteriorAI.py", line 527, in _createGame
    game.setExteriorZone(self.extZoneId)
  File "E:\Toontown\Toontown Fantasy\Toontown-Fantasy\toontown\cogdominium\DistCogdoMazeGameAI.py", line 43, in setExteriorZone
    self.createSuits()
  File "E:\Toontown\Toontown Fantasy\Toontown-Fantasy\toontown\cogdominium\DistCogdoMazeGameAI.py", line 67, in createSuits
    self.maxPickups = self._numSuits[0] * Globals.SuitData[0]['memos']
KeyError: 0
Press any key to continue . . .```

![image](https://user-images.githubusercontent.com/68795576/227755874-b6a779c4-6101-429f-9b32-dd994e0556c4.png)

doesn't seem to happen anymore


CASMCMS-9194: Update CFS export tools for v3 changes

The CFS export tools have been broken since the release of CFS v3, in two ways. First, the v3 endpoints added pagination, and the export tool did not handle this. Since the CLI defaults to using v2, the export tool still works a lot of the time, but on systems where there are enough CFS resources, attempts to list them using the v2 CLI will fail.
In addition, CFS sources are a new resource that have been added in CFS v3, and those were not exported by the tool.
This PR addresses both points. It also adds the collection of the CFS version as part of the export, since that could be useful for debugging.
I tested these changes on fanta and verified that they work as expected.
The backup files generated by the export tool are still in the same format that the import tool expects, so from that perspective there is no issue. However, the import tool does not handle CFS sources currently, and there may be some other v3-related issues I need to accommodate. That work for the import tool will be done with CASMCMS-8793. The changes in this PR are useful even without those changes, which is why I am splitting them up.
Backports forthcoming.

/backport release/1.5


Add @jhowardmsft to maintainers

How convenient is it that we can finally all react ":tada:" on pull requests in a single click, right?

Ping @jhowardmsft :-)

Of course, merging if windowsTP4 job is green :trollface:

One nit/question, but LGTM. Congrats @jhowardmsft

Wasn't expecting to come into the office and see that this morning :dog: :smile: :uk:

@jhowardmsft congrats!


build: update azure-cli version

What problem does this PR solve?:
Fixes devkit build error here.
To workaround the issue I updated azure-cli version.
Which issue(s) does this PR fix?:

https://jira.d2iq.com/browse/D2IQ-NUMBER

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

This unblocks https://github.com/mesosphere/konvoy-image-builder/pull/1239


Can't resolve module using next-mf 7.0.8 and newest Nextjs 13.5+ v

Hi all!
I've tried to upgrade app to newest nextjs 13.5.5. My federated modules stopped working even when using dynamic and ssr: false.
The next build says it can not resolve module. Works fine with 13.4.12.

is it working with app router?

@adarsh-drishya I'm still on Pages one, so can't answer :)

done use dynamic, use react lazy for ssr imports that are lazy

is it working with app router @BohdanYavorskyi

nothing will work in app router - you can try low level api tho

done use dynamic, use react lazy for ssr imports that are lazy

it works before 13.5+.
Using 13.5v is shows you the error even when you're using next/dynamic

@ScriptedAlchemy it just says it has an error. Using 13.4.12 (latest before 13.5) works fine

Is that issue resolved? The issue still occurs in 13.5 and later versions.

https://github.com/module-federation/universe/pull/2002


Cleanup and documenation

Renamed prologue.md to prologue.haddock. It is a haddock file not a markdown one. Rephrase some parts.

Regenerate the module map and provide a command to do so again.

Fixed broken links.

Re-format some lines to 80 chars.

Make haddocks.sh report no ShellCheck errors.

Fix haddock contents redirection.

Nicer output in haddocks.sh.

Include test-suites, benchmarks, executables in haddock output.

The nicer output


Make snippet-extract take individual input/output files instead of a directory

This does not change the experience for package authors but allows for better integration with other tools that require sandboxing of file access.
rdar://103034070

[x] Added tests
[x] Ran the ./bin/test script and it succeeded
[x] Updated documentation if necessary

@swift-ci Please test

@swift-ci Please test

Hm... ./bin/test succeeds here locally. Going to run again in case it's a bot issue.

@swift-ci Please test

@swift-ci Please test

@swift-ci Please test


Fix parenthesis handling on ranges and type2

This PR fixes two different issues related to parenthesis usage:

Use of parenthesis in .size notations was broken (ex: uint .size (1))
Handling of Type2::ParenthesizedType was missing

I still need to write tests for these two, but I'm waiting for #91

I'm going to merge this so that the tests can be added in #112 because right now the two PRs depend on each other in some cases so I'd like to get this one merged first