wifi mesh node do not find parent (or root) (IDFGH-11009)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
4.4.3
Operating System used.
Linux
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
None
Development Kit.
Custom board
Power Supply used.
External 3.3V
What is the expected behavior?
With fixed root up and running I expect any node to find and connect to the mesh network directly (10-20s) on power on.
What is the actual behavior?
Node searches for root/parent but fail to find any and ends up throwing MESH_EVENT_NO_PARENT_FOUND after . After that, no new searches are performed and the node ends up dangling outside of the mesh network.
After power cycle and a fresh start may or may not resolve the issue.
Problem occurs approximately 50% of the times at startup.
I have also tried to delay application related tasks to not interfere at startup without any better result.
Steps to reproduce.

Startup of the root
wait until stable
power on mesh node
wait for node to startup ~10-15s

Init code for the wifi/mesh is identical to code in the ip_internal_example:
Debug Logs.
Log when failure occurs:
I (27) boot: ESP-IDF v4.4.3-1-gcc5640aef3 2nd stage bootloader
I (27) boot: compile time 08:15:55
I (27) boot: chip revision: 3
I (29) boot.esp32: SPI Speed      : 80MHz
I (32) boot.esp32: SPI Mode       : DIO
I (34) boot.esp32: SPI Flash Size : 16MB
I (36) boot: Enabling RNG early entropy source...
I (39) boot: Partition Table:
I (41) boot: ## Label            Usage          Type ST Offset   Length
I (44) boot:  0 otadata          OTA data         01 00 0000d000 00002000
I (48) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (52) boot:  2 coredump         Unknown data     01 03 00010000 00010000
I (56) boot:  3 reserved         Unknown data     01 fe 00020000 00020000
I (59) boot:  4 hwadmin          Unknown data     01 d0 00040000 00001000
I (63) boot:  5 az_cert          unknown          40 00 00041000 00001000
I (67) boot:  6 az_cert_key      unknown          40 01 00042000 00001000
I (70) boot:  7 az_ca_api        unknown          40 02 00043000 00003000
I (74) boot:  8 az_ca_blob       unknown          40 03 00046000 00003000
I (78) boot:  9 nvs_key          NVS keys         01 04 00049000 00001000
I (82) boot: 10 nvs              WiFi data        01 02 00050000 00020000
I (85) boot: 11 filesystem       Unknown data     01 81 00070000 00370000
I (89) boot: 12 factory          factory app      00 00 00400000 00400000
I (93) boot: 13 ota_0            OTA app          00 10 00800000 00400000
I (97) boot: 14 ota_1            OTA app          00 11 00c00000 00400000
I (100) boot: End of partition table
I (102) boot: Defaulting to factory image
I (105) esp_image: segment 0: paddr=00400020 vaddr=3f400020 size=be5a0h (779680) map
I (345) esp_image: segment 1: paddr=004be5c8 vaddr=3ffbdb60 size=01a50h (  6736) load
I (347) esp_image: segment 2: paddr=004c0020 vaddr=400d0020 size=25fa1ch (2488860) map
I (1099) esp_image: segment 3: paddr=0071fa44 vaddr=3ffbf5b0 size=053c4h ( 21444) load
I (1107) esp_image: segment 4: paddr=00724e10 vaddr=40080000 size=1d994h (121236) load
I (1149) esp_image: segment 5: paddr=007427ac vaddr=50000000 size=00010h (    16) load
I (1165) boot: Loaded app from partition at offset 0x400000
I (1165) boot: Disabling RNG early entropy source...
I (1171) psram: This chip is ESP32-D0WD
I (1171) spiram: Found 64MBit SPI RAM device
I (1171) spiram: SPI RAM mode: flash 80m sram 80m
I (1173) spiram: PSRAM initialized, cache is in low/high (2-core) mode.
I (1177) cpu_start: Pro cpu up.
I (1179) cpu_start: Starting app cpu, entry point is 0x400817bc
I (0) cpu_start: App cpu up.
I (1726) spiram: SPI SRAM memory test OK
I (1734) cpu_start: Pro cpu start user code
I (1734) cpu_start: cpu freq: 240000000
I (1734) cpu_start: Application information:
I (1735) cpu_start: Project name:     ElsIotModule
I (1738) cpu_start: App version:      1.3.0
I (1741) cpu_start: Compile time:     Jun 22 2023 15:11:27
I (1744) cpu_start: ELF file SHA256:  8d10c98a9ef7eb70...
I (1747) cpu_start: ESP-IDF:          v4.4.3-1-gcc5640aef3
I (1750) heap_init: Initializing. RAM available for dynamic allocation:
I (1754) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
I (1757) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM
I (1760) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM
I (1763) heap_init: At 3FFD11D0 len 0000EE30 (59 KiB): DRAM
I (1766) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (1769) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1772) heap_init: At 4009D994 len 0000266C (9 KiB): IRAM
I (1776) spiram: Adding pool of 4084K of external SPI memory to heap allocator
I (1780) spi_flash: detected chip: generic
I (1782) spi_flash: flash io: dio
I (1827) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.

I (39) boot: Partition Table:
I (41) boot: ## Label            Usage          Type ST Offset   Length
I (44) boot:  0 otadata          OTA data         01 00 0000d000 00002000
I (48) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (52) boot:  2 coredump         Unknown data     01 03 00010000 00010000
I (56) boot:  3 reserved         Unknown data     01 fe 00020000 00020000
I (59) boot:  4 hwadmin          Unknown data     01 d0 00040000 00001000
I (63) boot:  5 az_cert          unknown          40 00 00041000 00001000
I (67) boot:  6 az_cert_key      unknown          40 01 00042000 00001000
I (70) boot:  7 az_ca_api        unknown          40 02 00043000 00003000
I (74) boot:  8 az_ca_blob       unknown          40 03 00046000 00003000
I (78) boot:  9 nvs_key          NVS keys         01 04 00049000 00001000
I (82) boot: 10 nvs              WiFi data        01 02 00050000 00020000
I (85) boot: 11 filesystem       Unknown data     01 81 00070000 00370000
I (89) boot: 12 factory          factory app      00 00 00400000 00400000
I (93) boot: 13 ota_0            OTA app          00 10 00800000 00400000
I (97) boot: 14 ota_1            OTA app          00 11 00c00000 00400000
I (100) boot: End of partition table
I (102) boot: Defaulting to factory image
I (105) esp_image: segment 0: paddr=00400020 vaddr=3f400020 size=be5a0h (779680) map
I (345) esp_image: segment 1: paddr=004be5c8 vaddr=3ffbdb60 size=01a50h (  6736) load
I (347) esp_image: segment 2: paddr=004c0020 vaddr=400d0020 size=25fa1ch (2488860) map
I (1099) esp_image: segment 3: paddr=0071fa44 vaddr=3ffbf5b0 size=053c4h ( 21444) load
I (1107) esp_image: segment 4: paddr=00724e10 vaddr=40080000 size=1d994h (121236) load
I (1149) esp_image: segment 5: paddr=007427ac vaddr=50000000 size=00010h (    16) load
I (1165) boot: Loaded app from partition at offset 0x400000
I (1165) boot: Disabling RNG early entropy source...
I (1171) psram: This chip is ESP32-D0WD
I (1171) spiram: Found 64MBit SPI RAM device
I (1171) spiram: SPI RAM mode: flash 80m sram 80m
I (1173) spiram: PSRAM initialized, cache is in low/high (2-core) mode.
I (1177) cpu_start: Pro cpu up.
I (1179) cpu_start: Starting app cpu, entry point is 0x400817bc
I (0) cpu_start: App cpu up.
I (1726) spiram: SPI SRAM memory test OK
I (1734) cpu_start: Pro cpu start user code
I (1734) cpu_start: cpu freq: 240000000
I (1734) cpu_start: Application information:
I (1735) cpu_start: Project name:     ElsIotModule
I (1738) cpu_start: App version:      1.3.0
I (1741) cpu_start: Compile time:     Jun 22 2023 15:11:27
I (1744) cpu_start: ELF file SHA256:  8d10c98a9ef7eb70...
I (1747) cpu_start: ESP-IDF:          v4.4.3-1-gcc5640aef3
I (1750) heap_init: Initializing. RAM available for dynamic allocation:
I (1754) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
I (1757) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM
I (1760) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM
I (1763) heap_init: At 3FFD11D0 len 0000EE30 (59 KiB): DRAM
I (1766) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (1769) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1772) heap_init: At 4009D994 len 0000266C (9 KiB): IRAM
I (1776) spiram: Adding pool of 4084K of external SPI memory to heap allocator
I (1780) spi_flash: detected chip: generic
I (1782) spi_flash: flash io: dio
I (1827) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (1844) spiram: Reserving pool of 24K of internal memory for DMA/internal allocations
I (2427) [main, 256]: Running partition, label: factory, type: 0x0, subtype: 0x0, address: 0x400000
I (2490) [main, 115]: efuse block [0, 0, 192]: 00 00 00 00 E4 D9 DB 93  17 44 57 00 00 A2 00 00  32 09 00 00 00 00 10 00
I (2493) [main, 115]: efuse block [1, 0, 256]: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
I (2549) [main, 115]: efuse block [2, 0, 256]: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
I (2552) [main, 115]: efuse block [3, 0, 256]: 00 00 00 00 00 00 00 00  00 00 42 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
I (4611) [mesh_wifi, 330]: mesh_wifi_init(root=false)
I (4627) wifi:wifi driver task: 3ffde190, prio:23, stack:6656, core=1
I (4639) system_api: Base MAC address is not set
I (4639) system_api: read default base MAC address from EFUSE
I (4663) wifi:wifi firmware version: 8cb87ff
I (4664) wifi:wifi certification version: v7.0
I (4665) wifi:config NVS flash: enabled
I (4665) wifi:config nano formating: disabled
I (4667) wifi:Init data frame dynamic rx buffer num: 64
I (4669) wifi:Init management frame dynamic rx buffer num: 64
I (4671) wifi:Init management short buffer num: 32
I (4675) wifi:Init static tx buffer num: 8
I (4677) wifi:Init tx cache buffer num: 64
I (4678) wifi:Init static rx buffer size: 1600
I (4680) wifi:Init static rx buffer num: 8
I (4682) wifi:Init dynamic rx buffer num: 64
I (4684) wifi_init: tcpip mbox: 32
I (4736) wifi_init: udp mbox: 6
I (4737) wifi_init: tcp mbox: 6
I (4737) wifi_init: tcp tx win: 5840
I (4737) wifi_init: tcp rx win: 5840
I (4739) wifi_init: tcp mss: 1400
I (4741) wifi_init: WiFi/LWIP prefer SPIRAM
I (4744) wifi:Set ps type: 0

I (4745) phy_init: phy_version 4670,719f9f6,Feb 18 2021,17:07:07
I (4944) wifi:mode : sta (44:17:93:db:d9:e4) + softAP (44:17:93:db:d9:e5)
I (4945) wifi:enable tsf
I (4947) wifi:Total power save buffer number: 4
I (4949) wifi:Init max length of beacon: 752/752
I (4950) wifi:Init max length of beacon: 752/752
I (5012) mesh: <nvs>read layer:0, err:0x1102
I (5017) mesh: <nvs>read assoc:0, err:0x1102
I (5127) wifi:Set ps type: 0

I (5155) mesh: [CONFIG]fail to set node, err:0x4008
I (5155) [mesh_wifi, 413]:  - router_ssid       : XXXXXXX
I (5155) [mesh_wifi, 414]:  - router_password   : XXXXXXXXXX
I (5158) [mesh_wifi, 415]:  - mesh_id           : XXXXXX
I (5161) [mesh_wifi, 416]:  - mesh_password     : XXXXXXXXXXXXXXX
I (5189) wifi:Total power save buffer number: 4
W (5654) wifi:age timer already deinit
I (5656) wifi:mode : sta (44:17:93:db:d9:e4)
I (5679) mesh: <MESH_NWK_LOOK_FOR_NETWORK>need_scan:0x3, need_scan_router:0x0, look_for_nwk_count:1
I (5685) [node_main, 74]: MESH is started
I (5694) [mesh_wifi, 422]: mesh starts successfully, heap:3932395, root fixed

I (7004) [mdf_mem, 199]: Free heap, internal current: 59099, minimum: 58907, total current: 3702135, minimum: 3700835

I (9168) mesh: [S1], fa:8f:ca:92:ea:09, channel:1, rssi:-92
I (9169) mesh: find root:ESPM_1E053C, root_cap:5(max:512), new channel:1, old channel:0
I (9170) mesh: [S3], fc:f5:c4:3c:fc:61, channel:1, rssi:-82
I (9172) mesh: [S4], 34:ab:95:5c:22:45, channel:1, rssi:-72
I (9175) mesh: [S5], fc:f5:c4:3c:f5:01, channel:1, rssi:-65
I (9178) mesh: [S8], c4:dd:57:b8:19:75, channel:1, rssi:-65
I (9180) mesh: [S9], c4:dd:57:b8:18:f1, channel:1, rssi:-67
I (9183) mesh: [S10], c4:dd:57:b8:19:29, channel:1, rssi:-67
I (9186) mesh: [S11], fc:f5:c4:3c:fc:0d, channel:1, rssi:-60
I (9188) mesh: [S12], fc:f5:c4:3c:f4:81, channel:1, rssi:-82
I (9191) mesh: [S15], fc:f5:c4:3c:fb:a9, channel:1, rssi:-55
I (9194) mesh: [S16], 84:1e:a3:47:84:b6, channel:1, rssi:-84
I (9196) mesh: [S17], fc:f5:c4:3c:f4:89, channel:1, rssi:-76
I (9199) mesh: [FIND][ch:0]AP:30, otherID:0, MAP:1, idle:0, candidate:0, root:1[4d:08:40:2c:75:09][FIXED-ROOT]
I (9204) mesh: [FIND:1]find a network, channel:1, cfg<channel:0, router:, 4d:08:40:2c:75:09>

I (9208) [node_main, 269]: MESH_EVENT_FIND_NETWORK
I (9209) wifi:mode : sta (44:17:93:db:d9:e4) + softAP (44:17:93:db:d9:e5)
W (9215) wifi:<MESH AP>adjust channel:1, secondary channel offset:1(40U)
I (9218) wifi:Total power save buffer number: 4
I (9474) mesh: [SCAN][ch:1]AP:8, other(ID:0, RD:0), MAP:1, idle:0, candidate:1, root:1, topMAP:0[c:0,i:0][4d:08:40:2c:75:09][FIXED-ROOT]<>
I (9476) mesh: 7369[selection]try rssi_threshold:-78, backoff times:0, max:5<-78,-82,-85>
I (9480) mesh: [DONE]connect to parent:ESPM_1E053C, channel:1, rssi:-38, c4:de:e2:1e:05:3d[layer:1, assoc:4], my_vote_num:0/voter_num:0, rc[00:00:00:00:00:00/-120/0]
I (9487) mesh: set router bssid:d5:32:80:f0:31:fd
I (9517) wifi:new:<1,1>, old:<1,1>, ap:<1,1>, sta:<1,1>, prof:1
I (9517) wifi:state: init -> auth (b0)
I (9529) wifi:state: auth -> assoc (0)
I (9548) mesh: <MESH_NWK_MIE_CHANGE><><><><ROOT ADDR><><><>
I (9548) wifi:I (9549) mesh: <MESH_NWK_ROOT_ADDR>from assoc, layer:2, root_addr:c4:de:e2:1e:05:3d, root_cap:5
state: assoc -> run (10)
I (9552) wifi:connected with ESPM_1E053C, aid = 2, channel 1, 40U, bssid = c4:de:e2:1e:05:3d
I (9556) wifi:security: Open Auth, phy: bgn, rssi: -39
I (9562) wifi:pm start, type: 0

I (9563) mesh: <MESH_NWK_ROOT_ADDR>idle, layer:2, root_addr:c4:de:e2:1e:05:3d, conflict_roots.num:0<>
I (9565) [node_main, 97]: <MESH_EVENT_ROOT_ADDRESS>root address:c4dee21e053d
I (9565) mesh: [scan]new scanning time:600ms, beacon interval:300ms
I (9584) mesh: 2004<arm>parent monitor, my layer:2(cap:16)(node), interval:9071ms, retries:1<normal connected>
I (9585) [mesh_netif, 469]: It was a wifi station removing stuff
I (9588) [mesh_netif, 84]: Receiving task started
I (9637) wifi:AP's beacon interval = 102400 us, DTIM period = 1
I (9641) [node_main, 88]: <MESH_EVENT_PARENT_CONNECTED>
I (9642) [node_main, 110]: MESH_EVENT_TODS_STATE toDS=1
I (9645) [node_main, 97]: <MESH_EVENT_ROOT_ADDRESS>root address:c4dee21e053d

More Information.
Using a mesh network system and have a custom software based on the ip_internal_network example (with a lot of application additions).
Root node is fixed!
ESP32-WROVER 16MB flash / 4MB ram
running on both cores
wifi module assigned to core 1
I also tried to modify the original example project with same changes we have in our project (sdkconfig):

enable SPIRAM
dual core
wifi on core 1
other tuning of SPIRAM usage
None of the changes in the example project caused similar behavior. The example project connect to my projects root 100% of the startups.

I have not tested more recent version of esp-idf since it breaks my project.

Now tested with es-idf 4.4.5. Same result as for 4.4.3

after reset of node it fails to find network 50% of the times
ip_internal_network example combined with sdkconfig from my project connects well
yes, all nodes except root (root gets esp_mesh_set_type(MESH_ROOT)
I do not call esp_mesh_set_self_organized as documentation says it is enabled by default

@ClaesIvarsson  Can you provide a demo for us to reproduce this issue?

@ClaesIvarsson Can you provide a demo for us to reproduce this issue?

Yes, I can demonstrate the problem, Maybe Friday at 10 CET is ok ? Please let me know and I will provide an invitation.

@ClaesIvarsson Can you provide a demo for us to reproduce this issue?

Yes, I can demonstrate the problem, Maybe Friday at 10 CET is ok ? Please let me know and I will provide an invitation.

Do you mean that you want to demonstrate your problem online? Actually, we hope you can provide a reproduce code so that we can test and debug it locally.

Well,

@ClaesIvarsson Can you provide a demo for us to reproduce this issue?

Yes, I can demonstrate the problem, Maybe Friday at 10 CET is ok ? Please let me know and I will provide an invitation.

Do you mean that you want to demonstrate your problem online? Actually, we hope you can provide a reproduce code so that we can test and debug it locally.

Maybe a misunderstanding from my side!
I am not sure I am allowed to share the existing code, will have to check how in such case.
To strip it down to something shareable would be rather large effort that I am not able to fit in my schedule right now. I will check how to proceed with my manager.

Is there anyway to get more debug information out from the mesh module ? Is there any information the provided logs that gives any hints on the problem at hand ?
Any explanation why the scanning of network doesn't restart after failure of finding a network ?
It looks like there is a full scan of WiFi networks at startup but that only happens once ... is this true ?

I have spent a lot of time trying to find a reason for this issue to happen.
My thesis has been that there is some kind of memory corruption, but I have not been able to find a pattern to follow.
I can see that some builds are more prune to fail than others. I have used git bisect function to try to find a commit that cause the issue. But my conclusion is that this happens on almost versions of my software since I introduced the NAT feature based on the ip_internal_network. If present, mainly depends on the number of tests performed.
I created a script that restart the node and waits for the event MESH_EVENT_PARENT_CONNECTED (with a timeout of 90s) and repeat this for N time (I ran my tests with 20 repetitions).

I have spent a lot of time trying to find a reason for this issue to happen. My thesis has been that there is some kind of memory corruption, but I have not been able to find a pattern to follow.
I can see that some builds are more prune to fail than others. I have used git bisect function to try to find a commit that cause the issue. But my conclusion is that this happens on almost versions of my software since I introduced the NAT feature based on the ip_internal_network. If present, mainly depends on the number of tests performed. I created a script that restart the node and waits for the event MESH_EVENT_PARENT_CONNECTED (with a timeout of 90s) and repeat this for N time (I ran my tests with 20 repetitions).

I noticed the same problem when using NAT with this same example (ip_internal_network). Yesterday I opened an issue, maybe it's the same problem: #12431

I noticed this problem when using NAT with this same example (ip_internal_network). Yesterday I opened an issue, maybe it's the same problem: #12431

We tried the example with minor adjustments used as node in the mesh network (not root) and didn't see this behavior at that point, but maybe we didn't test it enough times. I will try to do this test again with more repetitions.

I believe I have found the root cause for this problem!
The mesh password variable stored in the cfg structure was not null terminated when my password was copied. It rely on the fact that the cfg.mesh_ap.password buffer is initialized with zero value.
After adding a command to set the complete cfg structure to zero, I got rid of this problem.
I think it would be good if all mesh network examples are updated with this.
    mesh_cfg_t cfg; /*  = MESH_INIT_CONFIG_DEFAULT(); */
    memset(&cfg, 0, sizeof(mesh_cfg_t);
    cfg.crypto_funcs = &g_wifi_default_mesh_crypto_funcs;

or at least make sure that
buffers in the structure without length information is also terminated properly
cfg.router.password[sizeof(CONFIG_MESH_ROUTER_PASSWD)] = '\0';
cfg.mesh_ap.password[sizeof(CONFIG_MESH_AP_PASSWD)] = '\0';

I believe I have found the root cause for this problem!
The mesh password variable stored in the cfg structure was not null terminated when my password was copied. It rely on the fact that the cfg.mesh_ap.password buffer is initialized with zero value. After adding a command to set the complete cfg structure to zero, I got rid of this problem. I think it would be good if all mesh network examples are updated with this.
    mesh_cfg_t cfg; /*  = MESH_INIT_CONFIG_DEFAULT(); */
    memset(&cfg, 0, sizeof(mesh_cfg_t);
    cfg.crypto_funcs = &g_wifi_default_mesh_crypto_funcs;

or at least make sure that buffers in the structure without length information is also terminated properly
cfg.router.password[sizeof(CONFIG_MESH_ROUTER_PASSWD)] = '\0';
cfg.mesh_ap.password[sizeof(CONFIG_MESH_AP_PASSWD)] = '\0';

@ClaesIvarsson Thanks for your update, we will consider to update it.

@ClaesIvarsson
Why the IDF official example can work well ?  Is there anything special about your password？

@ClaesIvarsson Why the IDF official example can work well ? Is there anything special about your password？

I think it is luck or coincidence. The cfg variable in located on the stack which is uninitialized. My task is located in external ram, maybe this is the major difference.

@ClaesIvarsson @zhangyanjiaoesp
Thanks for the reporting and debuging.
We will close this issue, plz feel free to reopen is still have this issue.
Thanks.


Examples/docs for tributary.timer?

Right now struggling a bit with a visualization that doesn't work with a basic setup/d3.timer flow in tributary and would probably work with the timer framework that's available - but it's tough to track down examples of it / its api. Maybe I'm just missing a wiki page / example?

Thx!


 It is worthwhile to mention that the test only flashes the network core but leaves whatever application was flashed (and running) on the application core - therefore what software you end up running on the target depends on what you flashed previously to the application core.
My suspicious is that the network core is not fully independent from the application core, i.e. that the application core can cause delays on network core and thus ruin this test.

The failure strongly depends on k_cycle_get_32() output. Most k_cycle_get_32() calls contribute to small deviations but the initial seed value obtained in https://github.com/zephyrproject-rtos/zephyr/blob/68955ebdb6d8a9c6598810446fecc316209089e1/samples/kernel/metairq_dispatch/src/msgdev.c#L31-L47
strongly determines the outcome. Forcing state to 555810816 gives Queue full, event dropped! on almost every run. There are probably much worse seeds but 555810816 is first faulty seed value that I observed on nRF5340DK.

Crude patch to make the issue 100% reproducible on nrf52840dk_nrf52840 is to force RTC reset and wait for the counter to reach 0x2121.

This patch is not only crude but completely bogus. It causes really bad issues. Instead of directly triggering overflow, z_nrf_rtc_timer_trigger_overflow() should be called (ensure that CONFIG_NRF_RTC_TIMER_TRIGGER_OVERFLOW is enabled) - but then the issue does not reproduce on nrf52840dk_nrf52840.

I am observing very strange feedback loop. I am not sure how much it is affected due to stepping on breakpoint in debugger (i.e. if it is possible to observe such behavior without breakpoint - which does not stop RTC timer).
The feedback loop is in sys_clock_announce()
https://github.com/zephyrproject-rtos/zephyr/blob/b4538ffcdbfe6b0384601861c9469b93740f881a/kernel/timeout.c#L213-L226
the entry on list calls dev_timer_expired() indirectly via t->fn(t);. At the end of dev_timer_expired() new entry is scheduled in timeout_reset():
https://github.com/zephyrproject-rtos/zephyr/blob/b4538ffcdbfe6b0384601861c9469b93740f881a/samples/kernel/metairq_dispatch/src/msgdev.c#L71-L81
Once the control returns back to sys_clock_announce() loop, it picks up the just added timeout and executes it (i.e. calls dev_timer_expired()) and thus the cycle repeats. The feedback loop ends when m.seq < MAX_EVENTS is no longer true.
@andyross @nordic-krch Does this sound like some issue with sys_clock_announce() and/or the sample (dev_timer_expired()) or is it rather indicating something wrong with nrf_rtc_timer.c implementation?
#55729 prevents loss of messages even if the feedback loop hits (at most MAX_EVENTS messages are produced before the feedback loop breaks), but it does not address the feedback loop phenomena itself.

I guess it's plausibly due to your debug rig, but something is locking up interrupt deliver for an unacceptable amount of time.

The debug problem is that whenever you are stopped at breakpoint on nRF52840, the RTC timer (name is confusing, it is just 24-bit up-counter) keeps going. After resuming from any breakpoint, there is rather significant amount of pending ticks to announce (and these ticks do get announced - which triggers the feedback loop).

The root cause here is that the timer driver was eleven ticks behind the time it was supposed to have fired.

The eleven ticks was just one of the debug attempts I did... In reality, it is much worse! Hopefully now I see the full picture, so I am actually fine with simply letting the workaround in (as not much can really be done in tickless kernel case).
The first time a queue gets full depends heavily on RNG seed which is obtained from RTC COUNTER. This value is mostly the same on every reset (because RTC COUNTER gets reset when system resets - which happens to be non-problematic RNG seed), but sometimes (when this issue occurs) the value can get off the "expected constant" (largest deviation from the "expected constant" I have seen was 2593 counts).
When you get the problematic RNG seed, then if printk() is printing to 115200 baud UART, the printk() itself takes between 96 and 97 RTC counts. Note that the printk() is called in ISR, so there cannot be any tick interrupts. The lack of tick interrupts does not really matter, because the system is running TICKLESS_KERNEL. When dev_timer_expired() calls timeout_reset(), which in turn calls z_add_timeout(), the curr_tick is already at least 96 ticks behind! Considering that metairq_dispatch sample adds timeouts in range [0; 7] ticks - it is inevitable that at least 13 next dev_timer_expired() will happen back-to-back in sys_clock_announce() loop. And when the loop finishes, there will be even more unannounced ticks - so the first time k_msgq_put() in dev_timer_expired() fails (due to lack of space in queue), snowballs the remaining events to also fail in the feedback loop.

@andyross Could you please comment on following silly example from kernel perspective:
struct k_timer timer, timer2;
uint32_t cycles_before, cycles_after;

void expiry_fn2(struct k_timer *timer)
{
	cycles_after = k_cycle_get_32();
	printk("%d - %d = %d\n", cycles_after, cycles_before, cycles_after - cycles_before);
}

void expiry_fn(struct k_timer *timer)
{
	cycles_after = k_cycle_get_32();
	printk("%d - %d = %d\n", cycles_after, cycles_before, cycles_after - cycles_before);
	k_busy_wait(5 * 1000 * 1000);
	cycles_before = k_cycle_get_32();
	k_timer_start(&timer2, K_MSEC(10000), K_NO_WAIT);
}

void main(void)
{
	k_timer_init(&timer, expiry_fn, NULL);
	k_timer_init(&timer2, expiry_fn2, NULL);
	cycles_before = k_cycle_get_32();
	k_timer_start(&timer, K_MSEC(10000), K_NO_WAIT);
}

The expectation would be that the two lines printed by this silly example would have relatively small difference. However, on nRF52840DK, this gives:
*** Booting Zephyr OS build zephyr-v3.3.0-1145-g063c03948088 ***
339821 - 12140 = 327681
667500 - 502566 = 164934

which means that the k_timer_start() inside fires before its timeout expires. It all boils down to what is the base - if you consider the expiry_fn() entry as the base, then ktimer2 expiry time is expected. However, if you consider the time when k_timer_start() was called as the base, then it is wrong.
The above silly example seriously exaggerates what is happening in metairq sample.

And when they don't, it tries to do the right thing anyway by "faking" the current time. But nothing can really recover latency after a late interrupt; it's gone.

Yes, nothing can recover latency. But Zephyr tries to recover the latency - by shortening the relative timeouts scheduled in timeout handlers executed from sys_clock_announce(). Question is should such shortening take place?
Duration shortening that happens when sys_clock_announce() is running late effectively violates
https://github.com/zephyrproject-rtos/zephyr/blob/ec6122cd98beafe05a0910bac2df09f1688ef4d9/doc/kernel/services/timing/timers.rst#L55-L60
In the silly example, K_MSEC(10000) was requested, but the timer effectively fired K_MSEC(5000) after k_timer_start was called on timer2.
Or is it that the guarantee is only about the synthesized tick counter value and not about K_MSEC(x) representing x real-world milliseconds (within the source clock accuracy)?

But isn't this a wrong approach of those algorithms? Shouldn't they use a periodic timer instead?

The periodic behavior of k_timer is precisely the use case in question!  k_timer is based on the timeout mechanism we're discussing, and it doesn't "lose time" due to delayed interrupts because of this feature.  I buy that this isn't without controvery: this isn't the first time we've had a fight over how "current time" is exposed to timeout callbacks and the fact that in extreme conditions like this it can show a mismatch vs. the free-running cycle counter.  But on balance, I think we've made the right choice here.


ACPI: Fix full initialisation on several platforms

Full ACPI init was failing on many platforms. Some of the most critical issues were:

Attempt to write read-only memory (fixed by ACPICA module update)
Too small k_malloc heap for qemu_x86_64, up_squared and intel_ehl_crb
Attempt to register custom ACPICA system memory handler without unregistering default one
Incorrect PCI bus ACPI object name for up_squared and qemu_x86_64

I dropped the last commit which enabled PRT for EHL. This is so that the entire PR can potentially be backported (as it doesn't change any default features). PRT on EHL seems to work at least for the UART, but @finikorg reported that there's something wrong with SMBus, so this needs to be independently investigated.

The CI failures are not related to this PR, and are caused by #64557

The CI failures are not related to this PR, and are caused by #64557

Correction, the log_benchmark test is failing because of the above. There are other qemu test failures too, but they are all passing when I try to run them locally, so not sure what's going on.

ACPI full init is required only if DSDT support required. It is not required for EHL and QEMU since they need not required to retrieve interrupt routing table etc. from ACPI and this support added in Zephyr for later platform like RPL and ADL.

ACPI full init is required only if DSDT support required. It is not required for EHL and QEMU since they need not required to retrieve interrupt routing table etc. from ACPI and this support added in Zephyr for later platform like RPL and ADL.

People can still change the defaults when building applications based on these boards. Anyway, I'm not changing these defaults in this PR, but just making sure a full ACPI init can work on these platforms.
What's confusing to me however, is how the full init was working on any platform (even RPL or ADL), considering the read-only memory write issue and ACPICA error status return for trying to register a system memory handler. Those should fail on any platform, as far as I can see.
As for enabling the full init and PRT support on more platforms (which should be done in a separate PR), at least for qemu_x86_64 this is important so that we increase CI coverage. qemu is also the only platform with an integrated debugger, so it's often easiest & fastest to fix issues there, before trying to debug them on real HW.

@nashif @tbursztyka @finikorg @najumon1980 CI is passing now, so from my perspective this would be ready to go in.


Temporal SDK crash

Expected Behavior
Concurrent accesses to the maps need to be protected
Actual Behavior
Concurrent access to the map is causing SDK to crash, leading to a stuck client.
Steps to Reproduce the Problem
We are unaware of specific  ways to reproduce the issue. This happens intermittently and when it does, the impact is catastrophic as it impacts all the workflows executing on the runner.
We are using Temporal (and Temporal SDK) to develop a Serverless Workflow based orchestrator. We encountered this issue when a parent workflow triggers a large number of child workflows (Ex: ForEach statement in SWF). During the execution of the child, we are getting stack-traces like below
fatal error: concurrent map writes

goroutine 84634 [running]:
go.temporal.io/sdk/internal.(*commandsHelper).addCommand(0xc00211ed20, {0x31d45e0, 0xc0011bbfe0})
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_command_state_machine.go:958 +0x1eb
go.temporal.io/sdk/internal.(*activityCommandStateMachine).cancel(0xc00100d0e0)
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_command_state_machine.go:591 +0x7f
go.temporal.io/sdk/internal.(*commandsHelper).requestCancelActivityTask(0x31a57a0?, {0xc0013a0580?, 0x132eb00?})
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_command_state_machine.go:1020 +0x39
go.temporal.io/sdk/internal.(*workflowEnvironmentImpl).RequestCancelActivity(0xc00111f080, {{0xc0013a0580?, 0x2705c20?}})
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_event_handlers.go:663 +0x38
go.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteActivity.func3({0xc000a74fb8?, 0xc000615f28?}, 0x0?)
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/workflow.go:622 +0x75
go.temporal.io/sdk/internal.(*channelImpl).Close(0xc002418040?)
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_workflow.go:887 +0x97
go.temporal.io/sdk/internal.(*cancelCtx).cancel(0xc002418000, 0x0, {0x31a57a0, 0xc0000bfd40})
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/context.go:333 +0xff
go.temporal.io/sdk/internal.(*cancelCtx).cancel(0xc002c6dc20, 0x1, {0x31a57a0, 0xc0000bfd40})
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/context.go:340 +0x1e2
go.temporal.io/sdk/internal.WithCancel.func1()
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/context.go:196 +0x30
runtime.Goexit()
	/usr/local/go/src/runtime/panic.go:540 +0x1bb
go.temporal.io/sdk/internal.(*coroutineState).exit.func1({0xc000ac4000?, 0x3196760?}, 0xc000a58101?)
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_workflow.go:992 +0x17
go.temporal.io/sdk/internal.(*coroutineState).initialYield(0xc00211edc0, 0x3, {0xc000a58120, 0x19})
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_workflow.go:914 +0x89
go.temporal.io/sdk/internal.(*coroutineState).yield(...)
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_workflow.go:923
go.temporal.io/sdk/internal.(*channelImpl).Receive(0xc000f2e510, {0x31c5740, 0xc0015b28a0}, {0x0, 0x0})
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_workflow.go:715 +0x237
go.temporal.io/sdk/internal.(*futureImpl).Get(0xc0000ce3c0, {0x31c5740, 0xc0015b28a0}, {0x2718020?, 0xc000137080})
	/go/pkg/mod/go.temporal.io/sdk@v1.23.1/internal/internal_workflow.go:318 +0x66
gitlab.eng.vmware.com/core-build/swf-runtime/pkg/runtime.(*ForEach).runBatch(0x7?, {0x31c5740, 0xc0015b28a0}, {0xc000972000?, 0xc000aac501?, 0xc000844ba0?}, {0xc0007f9000?, 0xc00099f758?, 0x2c179b4?}, {0x2c00278, ...}, ...)
	/work/pkg/runtime/foreach.go:153 +0x215

The snippet of the code for foreach.go that's relevant to this is given below

futures := lo.Map(list, func(input map[string]any, i int) workflow.Future {
		future, settable := workflow.NewFuture(ctx)
		workflow.Go(ctx, func(ctx workflow.Context) {
			config := &RunActionConfiguration{
				Actions:        actions,
				Mode:           mode,
				Params:         input,
				Workflow:       configuration.Workflow,
				EnvironmentID:  configuration.EnvironmentID,
				WorkflowURL:    configuration.WorkflowURL,
				OrganizationID: configuration.OrganizationID,
			}

			settable.Set(f.actionsRunner.Run(ctx, config))
		})
		return future
	})

	var results []map[string]any

	var errList error
	for _, future := range futures {
		result := map[string]any{}
		if err := future.Get(ctx, &result); err != nil {
			errList = multierr.Append(errList, err)
		} else {
			results = append(results, result)
		}
	}

The line 153 in the foreach.go refers to future.Get()
Specifications

Version: 1.23.1
Platform: Linux/Kubernetes

Can you share what else is happening in your workflow? From the stack trace it looks like you are cancelling the context, but that is not in the code sample you shared.

runner_logs (2).txt
The workflow is invoking a collection of actionRunners, which in turn execute the ChildWorkflow. In the actionRunner code, I did find a deferred function that was cancelling the context.  This specific piece of code was inherited and I do not understand the purpose of the cancellation in this section of the code.
Also, at the end of the Child Workflow execution, the context is cancelled. This is because the runner doesn't distinguish between Child/Parent. At the end of every workflow execution the context is canceled.
Are these incorrect use of the SDK? I am attaching the complete logs.

Are you seeing this exception in your parent or child workflow?
As long as your following the constraints for programming in a workflow like determinism and not using goroutines/synchronization primitives, etc.

That should all be fine, this seems like an SDK bug. Are you using any disconnected contexts in this workflow?

Yes. Whenever a Parent/Child workflow is completed, it creates a Disconnected context. This is used to send events from the runner to an another entity to indicate the success/failure of the workflow.

ack most likely another instance of https://github.com/temporalio/sdk-go/issues/743

Hi Quinn,
Thank you so much for providing a likely diagnosis for this issue. I went through #743. People appear to have been struggling to reproduce the issue. In our case, it was observed very frequently (one in 3 experiments).
Setup:

Temporal (separate pods for history, matching, front-end, worker, web, etc) deployed Kubernetes Cluster deployed inside a VM
Task/Activity Pollers set to 32
We trigger a workflow, that invokes  128 child workflows in parallel using Foreach

Repeat the experiment. We hit it one out of 3 times
With the default poller setting of 2, we experienced it once in 10 experiments.
In our case, we didn't experience any significant loading as well.
Thanks,
Ashvin

Hi Ashvin
What has blocked us from debugging/fixing this issue is a lack of a full reproduction.
We understand it is some combination of child workflow, cancelation and possibly using a disconnected context leads to a race in the SDK

No unfortunately

@itendtolosealot I believe I found the issue. A potential work around until I can PR and release a fix would be to not use defers in workflows.


x/net/http2: 100-continue handling is case sensitive

What version of Go are you using (go version)?
This does not appear to be version dependent.  At least the go version that is the basis for several kubernetes / openshift releases are affected.
Does this issue reproduce with the latest release?
Yes, it appears to still be an issue: https://github.com/golang/net/blob/7805fdc37dc2b54b28b9d621030e14dcf1dab67c/http2/server.go#L2195
What operating system and processor architecture are you using (go env)?
N/A
What did you do?
According to https://datatracker.ietf.org/doc/html/rfc7231#section-5.1.1 and the similar issue https://github.com/envoyproxy/envoy/issues/20602 100-continue should not be a case sensitive value.
We have observed that different http clients, send different values for this.  In particular the JDK 11 httpclient sends 100-Continue: https://github.com/openjdk/jdk/blob/master/src/java.net.http/share/classes/jdk/internal/net/http/Exchange.java#L536
Thus when using the JDK client to interact with a go server based, this feature does not work as expected.

What did you expect to see?
Case insensitive handling of 100-continue.
What did you see instead?
The JDK client appears to hang as it gets a response it does not understand.

is this an issue for http1 and http2 or just http2?

cc @neild


Crash when creating Dataset from RaggedTensor, parallel function calls

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
Yes, see in colab below

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Docker image tensorflow/tensorflow:2.4.1-gpu-jupyter

TensorFlow installed from (source or binary):
binary

TensorFlow version (use command below):
2.4.1

Python version:
3.6.9

GPU model and memory:
Tesla V100 with 16GB

Describe the current behavior
Running the script in the colab below crashes:

at random times (but typically soon after launching the script)
most often with a segfault without further info, sometimes with a variety of error such as

tensorflow.python.framework.errors_impl.InvalidArgumentError: PartialTensorShape: Incompatible ranks during merge: 1 vs. 0
[[{{node TensorListConcatV2}}]] [Op:IteratorGetNext]
F tensorflow/core/framework/tensor_shape.cc:466] Check failed: end <= dims() (1 vs. 0)
invalid fastbin entry (free)
InvalidArgumentError: Trying to concat list with only uninitialized tensors but element_shape_except_first_dim_ is not fully defined: []
[[{{node TensorListConcatV2}}]]

Describe the expected behavior
No crashes, the expected output is
2
0
2
0
...
Standalone code to reproduce the issue
https://colab.research.google.com/drive/1TA2kXxM5mQXDKI5-mcY28IuTl4zO4Syv?usp=sharing
Other info
The problem does not occur if

the num_parallel_calls is set to 1
and/or
the ragged tensor is changed to an ordinary tensor
and/or
the tf.fill is changed to a constant tensor

@azertyken
Can you please refer to this resolved issue and let us know if it helps: link

@Saduf2019 I don't think the issue you refer to is similar to the one that I'm experiencing. Please note that most of the time, the script just segfaults and there is no error indication whatsoever.

@azertyken
I ran the code on tf 2.4 and nightly and do not see any errors on tf-nightly , nor does the colab crash, please find the gist here.

@Saduf2019 You're right, the errors are gone, thanks!

@ddeweerdt
Thank you for your update, please move the issue to closed status.


[CMake][MSVC] Disable permissive mode for MSVC builds

The C++ standard requires two-phase name resolution for templates.  By default, MSVC uses a non-standard name resolution, in which all names are looked up when a template is instantiated.  This has caused MSVC-specific compilation errors (example), which are quite difficult to debug.
This commit updates adds the /permissive- flag when building TVM with MSVC, disabling the non-standard name resolution.

I don't fully understand this patch, but seems that it solves a debuggability issue with MSVC. Is there any downsides for this change?

The only potential downside would be if we have any code that depends on the non-standard name resolution used by MSVC.  Most of the differences would result in compile-time failures, but there are some very weird edge cases that can occur.  (e.g. Suppose void func(int) is declared before the template, and void func(bool) is declared after the template but before the template instantiation.  Inside the template, a call to func(true) should resolve to void func(int), because name resolution happens when the template is parsed.  However, MSVC would silently and erroneously resolve it to void func(bool), because it does the name lookup at instantiation time.)
Even though two-phase name resolution has been the C++ standard as long as C++ has had a standard, MSVC still defaults to the non-standard name resolution for backwards compatibility with pre-C++98 code.  Their worry is that switching to standard name resolution would silently switch behavior that was developed and compiled only on MSVC.  Given that we primarily compile, test, and run on gcc/clang builds, I think this is very unlikely that we rely on MSVC's behavior, and any differences are most likely bugs in the MSVC build.

TL;DR: Very unlikely to introduce a bug, and any reliance on this behavior would have caused bugs in the gcc builds.

Got it. Yeah I do think it's very unlikely that people rely on MSVC-specific name resolution behavior. Thanks for explaining this to me!


Changed screenshots from PE Maker to new ones with TB BYOL

Changed screenshots in AWS installation instruction

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Keyboard remapping

Microsoft PowerToys version
0.57.2
Running as admin

[X] Yes

Area(s) with issue?
Keyboard Manager
Steps to reproduce
Remapped the windows key to Tilde in PowerToys because I needed it temporarily on a 60% compact keyboard. Went to change it back to the windows key after I got a new keyboard with both keys, but now the keyboard thinks the keys are the same, so if I make a change to the Tilde key it changes the function for the windows key, and vice versa. I have no other programs running that would have this effect on my keyboard. Even with all of the keyboard key remaps deleted in PowerToys, original settings are not reverted.
✔ Expected Behavior
Tilde back to Tilde
and
windows key back to windows key
❌ Actual Behavior
Tilde = windows key AND windows key = windows key
OR
Tilde = Tilde AND Windows = Tilde
Other Software
No response

Also, to be clear. I have both keyboards. Doesn't matter which is being used, keys have the same issue on both. So definitely software not hardware issue with new keyboard

Can I see either a report bug zip or a screenshot of the remapping?  /needinfo /reportbug

PowerToysReport_2022-05-25-11-46-45.zip

Figured out the issue.

sharpkeys may be a good solution here as it does a pure registry based solution.  it can't do shortcuts but can do key to key if memory serves


Init Scala3 compiler bridge

First step of https://github.com/sbt/sbt/issues/6080.
Minimal scala3-compiler-bridge
The scala3-compiler-bridge lives in its own module because it shares almost no sources with the Scala 2 compiler-bridge.
It is compiled by Scala 3, using sbt-dotty, because it depends on the Scala 3 compiler which is compiled in Scala 3.
As soon as sbt-dotty is merged into sbt, the scala3-compiler-bridge will be able to compiles itself.
This first minimal implementation is a direct translation of sbt-bridge in https://github.com/lampepfl/dotty:

https://github.com/lampepfl/dotty/blob/master/sbt-bridge/src/xsbt/CompilerInterface.java
https://github.com/lampepfl/dotty/blob/master/sbt-bridge/src/xsbt/CachedCompilerImpl.java

One minor difference is that it implements the new xsbti.compile.CompilerInterface2.
Add support for VirtualFile in Scala 3
This commit add support for VirtualFiles, that has recently been added to Zinc, to the scala3-compiler-bridge.
It fixes a bug that was reported in sbt/sbt#6007.
Also it fixes a bug in the returned CompileAnalysis by populating the SourceInfos.
The reported warnings were not returned back in sbt-dotty after a succefull compilation.
Improve DualLoader for Scala 3
Change the implementation of DualLoader to fix the LinkageError exception that occurs in the Scala 3 compiler when the xsbti.* classes are loaded.
The problem is that those classes are loaded by two different ClassLoaders: the sbt loader and the scala instance loader.
In the fixed implementation, the compiler classes are loaded by the DualLoader itself, which extends URLClassLoader, so that the subsequent xsbti.* classes can be loaded by the underlying dual loader which is the sbt loader.
See https://github.com/lampepfl/dotty/blob/master/sbt-bridge/src/xsbt/CompilerClassLoader.java for more details.

To validate this pull request against other sbt modules, please visit this link.

I changed the constructor of DualLoader. So It can possibly break the downstream projects that depend on it.
Yet, since DualLoader is in the internal package, these breakages are expected to be rare. There are at least two of them:

one in sbt that can easily be fixed by merging https://github.com/sbt/sbt/pull/6172
the other one is in sbt-dotty here. This breakage is actually unavoidable, but it can easily be fixed by deactivating the code on future sbt 1.5.0.

I found no usage of DualLoader in Bloop.

As intended, Mima complains about the hierachy and constructor of DualLoader:
 Error:   * the type hierarchy of class sbt.internal.inc.classpath.DualLoader is different in current version. Missing types {java.lang.ClassLoader}
Error:     filter with: ProblemFilters.exclude[MissingTypesProblem]("sbt.internal.inc.classpath.DualLoader")
Error:   * method this(java.lang.ClassLoader,scala.Function1,scala.Function1,java.lang.ClassLoader,scala.Function1,scala.Function1)Unit in class sbt.internal.inc.classpath.DualLoader does not have a correspondent in current version
Error:     filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("sbt.internal.inc.classpath.DualLoader.this")
Error:   * method this(java.lang.ClassLoader,scala.Function1,java.lang.ClassLoader,scala.Function1)Unit in class sbt.internal.inc.classpath.DualLoader's type is different in current version, where it is (Array[java.net.URL],java.lang.ClassLoader,scala.Function1,scala.Function1)Unit instead of (java.lang.ClassLoader,scala.Function1,java.lang.ClassLoader,scala.Function1)Unit
Error:     filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("sbt.internal.inc.classpath.DualLoader.this")

Also it complains that DifferentLoaders and NullLoader have disapperead:
Error:   * class sbt.internal.inc.classpath.DifferentLoaders does not have a correspondent in current version
Error:     filter with: ProblemFilters.exclude[MissingClassProblem]("sbt.internal.inc.classpath.DifferentLoaders")
Error:   * class sbt.internal.inc.classpath.NullLoader does not have a correspondent in current version
Error:     filter with: ProblemFilters.exclude[MissingClassProblem]("sbt.internal.inc.classpath.NullLoader")

That's because I found no usage in Zinc nor sbt nor Bloop and so I decided to remove them.
We can add them back later if it causes trouble in other downstream projects.

This scala3-compiler-bridge can be tried out in sbt by:

publishing Zinc and scala3-compiler-bridge locally: publishLocal; scala3CompilerBridge/publishLocal
fetching and publishing locally this version of sbt
using sbt-dotty to have a correct ScalaInstance
overriding scalaCompilerBridgeBinaryJar := None to bypass the Dotty sbt-bridge

// project/build.properties
sbt.version=1.5.0-SNAPSHOT

// project/plugins.sbt
addSbtPlugin("ch.epfl.lamp" % "sbt-dotty" % "0.4.6")

// build.sbt
lazy val root = project
  .in(file("."))
  .settings(
    scalaVersion := "3.0.0-M1",
    scalaCompilerBridgeBinaryJar := None
  )

One thing is that the current implementation does not handle suspendedUnits as the original Driver does here: https://github.com/lampepfl/dotty/blob/2f26fad3af54df76d5a898629d8d4f589c8083ce/compiler/src/dotty/tools/dotc/Driver.scala#L35-L49.
I intend to implement that very soon.

By the way, it looks like https://github.com/sbt/sbt/issues/6080 already mentioned "Merge sbt-bridge in Zinc." as the proposed implementation. I didn't notice this when I commented on that issue and thus didn't raise any objection to it at the time, sorry for that.

Thank you @smarter for telling this now, it's never too late... :)
So basically you want the compiler-bridge to be tied to the exact compiler version: sbt-bridge-3.0.0-M1, sbt-bridge-3.0.0-M2 and even for nightly versions. And of course we can implement that logic in sbt here so that we don't need sbt-dotty on future sbt versions.
It also means that the Zinc interface must be stable since the scala 3 compiler-bridge does not depend on the Zinc version anymore.

If we do change the API, we run into complicated bootstrapping issues: Scala 3 builds itself using sbt, but if sbt uses an incompatible bridge, we're stuck.

You can still maintain your own bridge in dotty for the sole purpose of bootstrapping. But I guess it is better to have one bridge for all purposes so that the Scala 3 and Zinc API converge.

By contrast, the zinc APIs used in the compiler bridge are pure Java APIs, and an effort has been made recently to make it possible to evolve them without breaking existing users of these APIs (#829, #872), the exposed API is also much smaller than the entire Scala 3 compiler jar, so the effort to keep it stable should be much smaller.

Right but the old CompilerInterface is already not precise enough for Zinc to work properly. Not only some new features (remote cache) are not supported but also some irreversible bugs where inserted, fixed for CompilerInterface2 but not for CompilerInterface.
Here the proposed implementation does extend the new CompilerInterface2 and fixes the aforementioned bugs. I would very much like to see those changes in the current sbt-bridge. But how far are we to sbt 1.4 in dotty?
(for compatibility with sbt 1.3 we can maintain both implementations but I doubt it is necessary)

I also see that your PR rewrote the bridge in Scala: we explicitly rewrote it in Java back in lampepfl/dotty#5596 to avoid complicated bootstrapping issues

So you expect the binary compatibility between two consecutive versions of Scala 3 to be broken?
I propose the following:

We upgrade dotty to sbt 1.4 (@smarter how hard would this be?)
I translate this implementation of CompilerInterface2 to Java and I move it to dotty. Part of the implementation could go to dotty internals actually, it would lighten the bridge and clarify the API. (we can maintain the old implementation if necessary which is not buggy in sbt 1.3)
The fix in DualLoader can be merged into Zinc so that we don't need the ugly CompilerClassLoader for future versions of sbt.

But how far are we to sbt 1.4 in dotty?

Just merged: https://github.com/lampepfl/dotty/pull/10498 :)

(for compatibility with sbt 1.3 we can maintain both implementations but I doubt it is necessary)

I'm fine with dropping support for sbt 1.3.

So you expect the binary compatibility between two consecutive versions of Scala 3 to be broken?

Not now, but eventually with 3.T/4 yes: https://github.com/lampepfl/dotty/issues/10244, having the bridge compiled once and pushed to maven also means sbt doesn't have to compile it with each new compiler version, which means no complicated cache handling (I know mill at least struggled quite a bit to cache the bridge correctly)

I propose the following: [...]

Sounds good to me! Though the changes to DualLoader need to be carefully inspected since ClassLoader are always tricky business and the way they are cached in sbt/zinc is also quite intricate (and if classloader caching doesn't work right, it might still work but slow things done as the JIT ends up recompiling the same classes again and again, it helps to run the JVM with -verbose:class to verify that multiple calls to compile for example do not end up reloading the same compiler classes multiple times.)

There is still a problem actually which is that implementing CompilerInterface2 in dotty will break compatibility of future Scala 3 versions with sbt from 1.4.0 to 1.4.4 because of the current DualLoader.
So the first thing I'll do is to fix the DualLoader for sbt 1.4.5. Then I can implement CompilerInterface2 and try to find a way to deactivate it on sbt version < 1.4.5.

I think the goal is to release a 1.5.0 that is fully scala 3 compatible before the scala 3 release. It doesn't necessarily seem worth a lot of effort to get things working with all versions of 1.4.x.


statusText: "BAD REQUEST", status: 400

{"ok":false,"url":"https://api.linode.com/v4/linode/instances","status":400,"statusText":"BAD REQUEST","headers":{"access-control-allow-headers":["Authorization","Origin","X-Requested-With","Content-Type","Accept","X-Filter"],"access-control-allow-methods":["HEAD","GET","OPTIONS","POST","PUT","DELETE"],"access-control-allow-origin":"*","connection":"close","content-length":"114","content-type":"application/json","date":["Sat","20 Nov 2021 21:40:04 GMT"],"server":"nginx","x-accepted-oauth-scopes":"linodes:read_write","x-customer-uuid":"BE68B4AB-B6EF-476D-988DDF77FFCA79AC","x-frame-options":"DENY","x-oauth-scopes":"*","x-spec-version":"4.110.0"},"text":"{\"errors\": [{\"reason\": \"type is required\", \"field\": \"type\"}, {\"reason\": \"region is required\", \"field\": \"region\"}]}","data":"{\"errors\": [{\"reason\": \"type is required\", \"field\": \"type\"}, {\"reason\": \"region is required\", \"field\": \"region\"}]}","body":{"errors":[{"reason":"type is required","field":"type"},{"reason":"region is required","field":"region"}]},"obj":{"errors":[{"reason":"type is required","field":"type"},{"reason":"region is required","field":"region"}]}}


tuned cache strategy & cache persistance

Hello
Here are a set of patchs that add 2 majors features to carbon cache daemon, that improve performance and resiliance :

write cache memory to disk at shutdown and regular interval to minimize unflushed metrics lost if daemon is killed. The file is automatically loaded at startup
new tuned cache  write strategy that allow to focus flushing queues based on the size, name, age... Flushing occurs for a time or a number or percent of the queue length

This allow me to manage many queues and metrics without having a great storage backend (no more need of SSD storage)
This strategy consume more memory, but allow the system to be more responsive (less CPU & I/O)
See Git comments for some details
Also : A PR has been pushed on graphite-api GIT to add new APIs to dialog with this patch :
(https://github.com/graphite-project/graphite-web/pull/1277)

force to write cache memory to disk
force queue flushing of some queues matching a regexp

@llicour
Quite interesting patch,  thanks! Could you please explain how new cache strategy helps you with slower storage?

Ok, here is the big picture
When carbon cache receive more metrics it can process (that is the case if there is a slow storage backend), metrics are queued in memory (in a dictionnary structure) and dequeued by a thread, using a specific strategy
At this point, in memory, we have array of metrics, associated to a dictionnary key (the metric name)
Without dequeuing thread, system memory will be consumed until Out of memory (via swapping..)
Goal is to dequeing as fast as possible, considering the backend storage.
Dequeuing is done queue per queue, (a queue equivalent to a whisper db file .wsp)
There is a cost to process a whisper file. Using update_many() api allow to optimize I/O by updating DB file with many metrics in one operation
Consider that updating a whisper DB with 100 metrics can be equivalent than updating it with 1 metric, as 1 metric is only a few bytes
We can consider that for a specific hardware, the rate of queue/second is something that is fixed (it's depend of the hardware, and can be evaluated using the carbon metrics)
It's possible to limit this rate using the MAX_UPDATES_PER_SECOND, and decreasing this value allow to have a responsive system, as it consume less I/O and CPU
However, memory will growing, as we decrease the dequeing capability
As the rate queue/sec is fixed, if we want to limit memory consumption, we have to work on the rate metrics/sec
Especially with a slow storage (but not only), it's better to have many metrics per queue (whisper DB) to update (rather than few) as it optimize this rate
Native strategies are not (well) design for this

naive : it works randomly over the queue. Nothing to do with it
sorted : the list of queues in memory are sorted by the number of metrics. Excellent ! So the thread process first queues with lots of metrics (that is a good idea). Pb is that all the queues are processed (even with few metrics). So if you have many queues, it will take time (hours...) before to process all the queues.
max : this only select THE queue that contains the more metrics. Nice try. But if there is a lot of queues  in memory, the CPU is only busy to sort the queues (that takes seconds), for 1 whisper DB update. Unusable...

Idea of a new strategy was born here : Having a way to only select queues that worth to be flushed, by processing a 'sorted'  strategy only for the top 100 (for example)
Tuned strategy also allow to have more criteria to select queues
One of the drawback of this strategy is that queues can be keeped in memory for long time before beeing flushed (on my system, this can be days). CarbonLink queries can take care of this, so this is not a design problem
However, there is a problem if carbon-cache process is shutdown or system reboot, as all queues in memory will be lost. Also,during a clean shutdown, it can take a long time (hours) before all the queues to be flushed.
For these reasons, the patch also add a mecanism to flush the cache memory in a flat file (grep-able and awk-able) at shutdown, that is reloaded during startup. It takes only few seconds to create this file...
I also think that even with a speedy storage (SSD), it can be valueable to use this strategy (decreasing the MAX_UPDATES) to limit the micro-I/O that can be done on the SSD drive (that could tired the disk)
Hope these explanations (in my own forked english) could help you ...
Laurent

Thanks,  Laurent , excellent explanation!  Found your commit comments too. Will try your patch soon on our cluster.

ok.
So wait for few days, as i'll push commit that add metrics about tuned strategy, as well as graphite api that allow to modify strategie params online (without restarting cabon-cache)

I've just update the PR to include

new carbon.agents metrics that allow to follow 'tuned' strategy processing
new CarbonLink command that allow to change some carbon cache settings online

The following settings are supported :

CACHE_WRITE_STRATEGY
CACHE_WRITE_TUNED_STRATEGY_LARGEST
CACHE_WRITE_TUNED_STRATEGY_RANDOM
CACHE_WRITE_TUNED_STRATEGY_OLDEST
CACHE_PERSIST_INTERVAL
MAX_UPDATES_PER_SECOND
Sample :
curl localhost/metrics/set-param -X POST -d '{"key":"CACHE_WRITE_STRATEGY","value":"tuned"}'
curl localhost/metrics/set-param -X POST -d '{"key":"MAX_UPDATES_PER_SECOND","value":"1"}'

Note: changes are not persistant. If process is restarted, settings from configuration file will be restored

Hey guys, did this make it to the 0.9.13 tarballs released last week?

0.9.13 still not released, as far as I know. And this PR is not even in
milestone, so...
I tested cache persistence on my own branch (maybe I need to open source
it) but didn't test new strategy...
On Thu 20 Aug 2015 at 19:22 FRLinux<EMAIL_ADDRESS>wrote:

Hey guys, did this make it to the 0.9.13 tarballs released last week?
—
Reply to this email directly or view it on GitHub
https://github.com/graphite-project/carbon/pull/419#issuecomment-133081697
.

Thanks for the clarifications. Looks like the tarballs for webui and carbon are now gone from http://graphite.readthedocs.org/en/latest/releases/0_9_13.html

I think this is a very interesting patch but it really belongs in master, not the 0.9.x branch. @llicour would you be so kind as to rework this patch against master Carbon and Graphite-Web?

No problem for working on this, but i have to install a plateform from the master branch.
Could you clarify the roadmap about graphite/carbon branch ?
It seems to me that the 0.9.x branch is the current stable version (latest version published on pypi)
and that master branch is an experimental branch (a complete rewrite) that does not have been published yet
What is the roadmap of each of these branches (main features, next release dates...) ?
Is the 0.9.x branch a dead branch ?

The 0.9.x branch was forked a long time ago, I don't even remember why at this point (predates my involvement). We've nursed it along as the "stable" branch, mostly applying security patches & bugfixes and cherry-picked features from master deemed too important to skip. All new feature development should always be targeted at the master branch.
The Graphite Project is entirely run by volunteers. As such it's impossible (or at best, disingenuous) to give release dates in advance. My off-the-cuff answer to "when is the next release cut from master" would be something along the lines of "probably 3-6 months after our next (and final) release from 0.9.x", which should be any day or week now.

@llicour As per my comment in https://github.com/graphite-project/graphite-web/pull/1277, this looks really cool but please resubmit against master. Thanks! :heart:

Hi ...
When this will be merged to master. i checked in 9.15 but i dont see the changes .
please help with the info.

It was never merged to 0.9.x and an author does not submit it to master.
I personally using part of this patch in PROD and will port it to master soon.
Stay tuned. :)

Just a status update:

patch is quite harsh with twisted reactor. If flush or persisted cache save take more than 15-20 seconds reactor going a bit mad, and causing carbon cache reconnect with relay. Fixed it with increasing cache on relay on my working 0.9.x installation, but it's still quite scare.
porting this patch to master is also not straightforward. Tried that - bit no luck yet. Will try to create my own solution, somewhat later.

i tried to make the build on centOS 6, i get following error.
Compiling /builddir/build/BUILDROOT/graphite-web-0.10.0_rc1-1.x86_64/opt/graphite/webapp/graphite/remote_storage.py ...
SyntaxError: ('invalid syntax', ('/opt/graphite/webapp/graphite/remote_storage.py', 177, 13, ' for series in unpickle.loads(pickled_response)\n'))
plz help.

Sorry, @cchiremath, not really sure what you trying to compile. This patch is against 0.9.x branch, you can't apply it on a master branch - there are too many changes.


[WEBSITE] skip link a11y

Update documentation to have a skip link shortcut for better a11y.

Fixed nav style change to be a website only change.


Likes

This should be useful for implementing the comments because these can be stored also in the db and just like the two models (video and comment).
Like/dislike buttons are sending an asynchronous request (via jquery/ajax) in flask/media_server.py with a corresponding string indicating like or dislike. Flask receives them and updates the database num of likes (row) for the current video (column).
The video object stored in the database is stored in models.py. Other objects to be stored in the db could be added there too (don't forget to apply db migrations when making changes in the models).
Because the both media_server.py and models.py need a reference to the db -which is initialized within media_server.py- we create it in globalimports.py which we later import everywhere else to avoid circularity of SQLAlchemy's imports.
New dependencies where needed (flask-sqlalchemy, flask-migrate) which are listed in requirements.py.
Further improvements:
-limit even more the requests to the db by not committing on every like increment/decrement
-add an likes counter inside the video page
-style buttons

Hey, thanks for the PR, I'll review it soon! In the meantime, is it possible that you can provide a deployed version of your PR on something like Netlify? If not, no problem, I'll clone and try it out soon.

I'm afraid I'm not familiar with netlify so I'll leave that up to you.
What about the travis' tests? Seems that every recent pr fails due to not executing correctly tests.py and regardless of the pr itself. Am I right?


Make ErrorImpl public

Simple change to make ErrorImpl public.
I'd like to be able to use the Marker Information from the scanner error to provide users that are changing files the ability to check to see whether their yaml is valid and, if not, where the issue lies.  To do this i need the Marker information from the ScannerError in a structured format, so I can feed that into an editor.
This resolves https://github.com/dtolnay/serde-yaml/issues/78
I also have a PR out about getting access to the Marker for Scanner errors, which is also not public.

Ok, I've added two new structs to the Error which decouple the Error from yaml_rust in the future.  They are ErrorMarker which is just Marker, and ErrorPos which is a description with an optional ErrorMarker.
The ErrorPos struct is basically what I'm after, so I can get a nice location of where the error is happening (if one exists).
Unfortunately though, yaml_rust does not currently expose a way of getting the marker from a scan error, which I have submitted a PR for, but may not get accepted, so I've just pointed this PR to my branch.
But to make matters worse, the new version of yaml_rust escapes y as a key with quotes around (i.e, "y" instead of y) I think to align to the bool definition within yaml which breaks assertions and examples.

Please rebase on the master branch which includes the update to yaml-rust 0.4.0.

I've adjusted the from to be a private fn of ErrorMarker rather than implementing the From trait.  Is this enough to hide it?

I've updated it to be a bit cleaner and provide some more documentation around usage.

How come the column does not match what you see when you print the error?
extern crate serde_yaml;
use serde_yaml::Value;

fn main() {
    let err = serde_yaml::from_str::<Value>("@").unwrap_err();
    let marker = err.marker().unwrap();
    println!("{}", err);
    println!("line={} col={}", marker.line(), marker.col());
}

unexpected character: `@' at line 1 column 1
line=1 col=0

@dtolnay I'm not sure about why the marker is different from the error. I'll have a look at the yaml crate and see if I can pin that one down.
@killercup I'll make those adjustments and check that it formats it as you said.

I've pushed a new commit to the branch with some of those suggestions.
I've changed the name simply to Location and used column() rather than col.
With regards to the column not matching the error message, I've just added + 1 to what we get from the marker, as that is what they're doing when they print it out in the yaml crate.
Also added a test case for the None side of things.

I will get a release out tomorrow with this change and the update to yaml-rust 0.4.

Awesome!


Whitepaper: Emphasize the benefits of a dataspace vs. other approaches

“You know what? Let’s build a joint venture together.” Francesca proposes. “A non-profit that facilitates data exchange between food retail stores and chains and their suppliers, where everyone involved can exchange data in a controlled manner.

Here we can mention the alternative approaches and their pros and, especially, cons, which could be:

a central data lake
organizing the data exchange without a dataspace system by exchanging data directly basing on (paper) contracts
...

There we can add benefits of data space systems in general

decentralized data storage, on-demand data integration
policy management and policy enforcement
...
evtl. also additional options such as demand-driven application of data pre-processing services (e.g. anonymization)

"how to design a data space"
the whole document is about "why we need data spaces"
"dramatic arch" in storytelling; e.g. negative public examples whith data lakes


Feature request: Column display of menu

Right now it seems only possible to display a vertical selection menu, however it would be quite useful to have a columnar display option.
This could save some space on the screen for the preview and other components.

This is a good idea. Probably, it would be even better, to allow different positions and layouts for all components, but this would need a major rewrite of the code...


add support for spotfy

Go ahead, ban me if im being annoying with my PRs
Was discussed in #4
Spotfy™
Preview:
Main profile:

Small profile:

On the main profile screenshot it seems the pronouns aren't the same size as the existing text, has this been fixed by 41cb540?

On the main profile screenshot it seems the pronouns aren't the same size as the existing text, has this been fixed by 41cb540?

Yes, I forgot to update the preview, apologies.


Replace existing terminal with warp

Replace terminal with the warp terminal when it becomes available to linux users.

moved to private task list


Exemplary project for Arduino Uno board with qemu-system-avr support.

Exemplary project for Arduino Uno board with qemu-system-avr support.

This project bases on AVR_ATMega323_WinAVR example. The MCU and its port was changed. Make scripts ware extended support for qemu and visual code editor was added.
Test Steps
Go to FreeRTOS/Demo/AVR_ATMega328_ArduinoUno
make
make gdbserver
attach to gdbserver
Checklist:
I don't expect any surprises. I just added in in order to avoid braking many times opened doors.
Related Issue
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hello. Thank you for your reply.
I'm working on it.
There are some issues:

Unfortunately There is not enough flash memory in this MCU (32 kB) to
place everything. I will comment out some files and tests, or prepare two
targets.
I needed to make some changes in FreeRtos in order to compile. I can
prepare additional PR for FreeRtos in order to add this project.

diff --git a/include/deprecated_definitions.h
b/include/deprecated_definitions.h
index b046711a6..6e2a03721 100644
--- a/include/deprecated_definitions.h
+++ b/include/deprecated_definitions.h
@@ -50,9 +50,10 @@
#endif
#ifdef GCC_MEGA_AVR

#include "../portable/GCC/ATMega323/portmacro.h"

#include "../portable/ThirdParty/GCC/ATmega/portmacro.h"
#endif

#ifdef IAR_MEGA_AVR
#include "../portable/IAR/ATMega323/portmacro.h"
#endif
diff --git a/portable/ThirdParty/GCC/ATmega/portmacro.h
b/portable/ThirdParty/GCC/ATmega/portmacro.h
index 3e7714eb8..a6b501019 100644
--- a/portable/ThirdParty/GCC/ATmega/portmacro.h
+++ b/portable/ThirdParty/GCC/ATmega/portmacro.h
@@ -51,6 +51,7 @@ extern "C" {
#define portDOUBLE                  double
#define portLONG                    long
#define portSHORT                   int
+#define portBASE_TYPE               char
typedef uint8_t                     StackType_t;
typedef int8_t                      BaseType_t;
Pozdrawiam
Adam Kaliszan
wt., 11 kwi 2023 o 20:14 Aniruddha Kanhere @.***>
napisał(a):

Hello @adamkaliszan https://github.com/adamkaliszan,
You can find more information about the steps required to add your demo to
the community supported boards here
https://github.com/FreeRTOS/FreeRTOS/blob/main/FreeRTOS/Demo/ThirdParty/Template/README.md
.
These are the demos which are contributed by the FreeRTOS community
members.
If you need any help to create a PR in the community supported demos
repository, which @rawalexe https://github.com/rawalexe pointed to
above, please do not hesitate to reach out to us. We will be more than
happy to help you out.
Thank you again for taking the time to create a demo and to create a PR.
Best,
Aniruddha
—
Reply to this email directly, view it on GitHub
https://github.com/FreeRTOS/FreeRTOS/pull/985#issuecomment-1503871716,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH2XVLMY3T4IDWD2536OBZ3XAWNRBANCNFSM6AAAAAAWYEJKOY
.
You are receiving this because you were mentioned.Message ID:
@.***>


[#1909] [FOLLOWUP] improve(common): Avoid load missing key from system properties

What changes were proposed in this pull request?
Avoid load missing key from system properties
Why are the changes needed?
(Please clarify why the changes are needed. For instance,

If you propose a new API, clarify the use case for a new API.
If you fix a bug, describe the bug.)

Fix: Followup work of #1909
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Tested locally.
added jvm args -Dmissing.key=xxx -Drss.coordinator.app.expired=50000 -Drss.rpc.metrics.enabled=false
then, check the result.

@xianjingfeng Thanks for your review about my previous PR https://github.com/apache/incubator-uniffle/pull/1909
It can works well except that it set all system properties into config map.
It can be better with this improvement, PTAL.

I don't understand what the problem would be without this PR.

I don't understand what the problem would be without this PR.

https://private-user-images.githubusercontent.com/17329931/351759862-44997d45-8bcb-4ecd-9040-5ee435418565.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE5NzcxMjcsIm5iZiI6MTcyMTk3NjgyNywicGF0aCI6Ii8xNzMyOTkzMS8zNTE3NTk4NjItNDQ5OTdkNDUtOGJjYi00ZWNkLTkwNDAtNWVlNDM1NDE4NTY1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzI2VDA2NTM0N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQ2ZDI1NDVlYTJjZDFlNjhjNTUyOTZiNzlhZTBlM2JhZjFmYmZjN2JlMmJhMDMzZWJkYWUwMzc0MDFmOWUwZWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ia8pRsa2cj-d2somwnqZAmHxMR0HMy21KYN4BcBfKoY
All jvm args will be displayed in dasboard

@xianjingfeng Thanks for your review!


Sandbox not building with USE_ADD_SUBDIRECTORY

Hello,
When I try to run the sandbox with USE_ADD_SUBDIRECTORY set, I'm getting an error about the conanfile.txt not found, I'm not really good in conan to see where this would need to be fixed, as the conan_provider.cmake is simply using CMAKE_SOURCE_DIR which is the project source direct.
Any help is appreciated!
-- CMake-Conan: Installing single configuration Debug
-- CMake-Conan: conan install /Users/tomdebrouwer/Code/src/libraries/nmos-cpp/Sandbox/my-nmos-node -of=/Users/tomdebrouwer/Code/src/libraries/nmos-cpp/Sandbox/my-nmos-node/conan --profile:host=default;--profile:host=/Users/tomdebrouwer/Code/src/libraries/nmos-cpp/Sandbox/my-nmos-node/conan_host_profile;--profile:build=default;--build=missing
ERROR: Conanfile not found at /Users/tomdebrouwer/Code/src/libraries/nmos-cpp/Sandbox/my-nmos-node/conanfile.txt
CMake Error at /Users/tomdebrouwer/Code/src/libraries/nmos-cpp/Development/third_party/cmake/conan_provider.cmake:436 (message):
  Conan install failed='1'
Call Stack (most recent call first):
  /Users/tomdebrouwer/Code/src/libraries/nmos-cpp/Development/third_party/cmake/conan_provider.cmake:543 (conan_install)
  /Users/tomdebrouwer/Code/src/libraries/nmos-cpp/Development/cmake/NmosCppDependencies.cmake:19 (find_package)
  /Users/tomdebrouwer/Code/src/libraries/nmos-cpp/Development/CMakeLists.txt:29 (include)

-- Configuring incomplete, errors occurred!

The issue is that the CMake "top-level include" approach to using the Conan provider requires a top-level conanfile.(py,txt), it doesn't gather conanfiles from the subdirectories (rationale explained in https://github.com/conan-io/cmake-conan/issues/621). So I think the documentation for building the Sandbox/my-nmos-node needs to be clarified.
An alternative to using the Conan provider is to install the necessary libraries and then just use the generated toolchain.
Something like I do in garethsb/nmos-cpp-examples... maybe, untested:
cd Sandbox/my-nmos-node
mkdir build
cd build
conan install ../../Development -g CMakeToolchain --settings:all build_type=<Debug-or-Release> --build=missing --output-folder=conan
cmake .. \
  -DUSE_ADD_SUBDIRECTORY:BOOL=1
  -DCMAKE_BUILD_TYPE:STRING="<Debug-or-Release>" \
  -DCMAKE_TOOLCHAIN_FILE=conan/conan_toolchain.cmake
make

Thanks for the quick reaction.
For now I copied the conanfile to the toplevel directory which gets me through the conan step.


[Broken] Niconico - Long videos fail downloading

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dlc version 2020.10.26
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

$ youtube-dlc --ignore-config --verbose 'https://www.nicovideo.jp/watch/sm29303999'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--ignore-config', u'--verbose', u'https://www.nicovideo.jp/watch/sm29303999']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.10.26
[debug] Python version 2.7.18 (CPython) - Linux-5.4.0-52-generic-x86_64-with-Ubuntu-20.04-focal
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
[debug] Proxy map: {}
[niconico] sm29303999: Downloading webpage
[niconico] sm29303999: Downloading JSON metadata for h264_360p-aac_64kbps
[niconico] sm29303999: Downloading JSON metadata for h264_360p_low-aac_64kbps
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://pe03821af87.dmc.nico/vod/ht2_nicovideo/nicovideo-sm29303999_d12af73024e45c5410966649f9601391bdf4129f978a797c374c3bb3ce252d22?ht2_nicovideo=6-N0wawdNOqC_1604121810978.4nohp4qioy_qj1wb7_20e41xpuox7ii'
[download] Destination: 【東方】幻想万華鏡　第７話・巨大妖怪伝説の章【二次創作アニメ】-sm29303999.mp4
[download]  23.8% of 58.16MiB at 120.34KiB/s ETA 06:17[download] Got server HTTP error: Downloaded 14524539 bytes, expected 60982677 bytes. Retrying (attempt 1 of 10)...
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dlc/YoutubeDL.py", line 2003, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dlc/YoutubeDL.py", line 1861, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dlc/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dlc/downloader/http.py", line 349, in real_download
    establish_connection()
  File "/usr/local/bin/youtube-dl/youtube_dlc/downloader/http.py", line 114, in establish_connection
    raise err
HTTPError: HTTP Error 403: Forbidden

Description

Downloading long videos requires heartbeat signals. Here's the description found in the original issue on youtube-dl's repo:

As of today, Nicovideo's seem to implemented a new session system. After every few minutes, the url of the video changes. This causes a 403 Forbidden error on the old url, Causing the Downloader to fail.
A screenshot of part of a video session:
chrome 2017-10-25 10 43 35 396
Resuming the download works just for a few minutes before the session is dropped.
Note: I have only tested on a non-premium account. Im not sure if it applies to the permium accounts or not.

URL example: https://www.nicovideo.jp/watch/sm29303999

Same problem with https://www.youtube.com/watch?v=VI51y4dfrcM

Also returns bad url of formats for long youtube videos, https://www.youtube.com/watch?v=VI51y4dfrcM for example


HOTFIX: GlobalConfig updates

Coverage increased (+0.03%) to 93.712% when pulling d51c5393e759c4581698f934c17c9812229483a5 on RickCarlino:master into 1a35dfb136134d88b9d880a5a3bcedffa3f0b18f on FarmBot:master.


OT 3.0.3 still seeing -  Error: java.lang.RuntimeException: No renderer configured for com.vladsch.flexmark.ext.typographic.TypographicSmarts

Expected Behavior
Should not fail.
Actual Behavior
Builds Fail when certain characters are in CommonMark/markdown files.
If the following characters are in my MD file:
...
'
then I'm seeing the error:
Error: java.lang.RuntimeException: No renderer configured for com.vladsch.flexmark.ext.typographic.TypographicSmarts
Possible Solution
Steps to Reproduce

I found that any MD file I had that contained either ellipses or single quotes caused the error.

DITA-OT version: 3.0.3

Operating system and version: Mac OS X 10.13.3

How did you run DITA-OT? dita command

Transformation type:
HTML5, PDF

This issue is related to: https://github.com/dita-ot/dita-ot/issues/2866
However, that issue has been closed and I'm challenging that it should have been closed.
I'm using:
<topicref href="myfile.md" format="markdown" />

That issue was really only closed because it was moved to the repository dedicated to the Lightweight DITA plugin (yes, I know, that's a bit confusing) -- and should have been fixed with https://github.com/jelovirt/org.lwdita/issues/3
I'm not sure if that fix was bundled with 3.0.3 or not though.

Looks like there were 2 separate commits to address the issue in jelovirt/org.lwdita#3:

jelovirt/org.lwdita@2c01572 was included in org.lwdita v2.0.2, which was bundled with the DITA-OT 3.0.2 release.
This commit resolved the issue for the mdita format per https://github.com/dita-ot/dita-ot/issues/2866#issuecomment-371831669:

I got the same issue in 3.0.2 depending on the format flag.
When using format="mdita" in the map everything worked fine. When I used format="markdown" though, the same error as above appeared.

jelovirt/org.lwdita@da97502 was added later to sync the same change to the markdown format, but there has been no release of the plug-in since, and thus DITA-OT 3.0.3 does not include this fix.

@jelovirt It would be good to package the latter change into a v2.0.3 release of org.lwdita along with the changes for jelovirt/org.lwdita#8 and bundle the result with the DITA-OT 3.0.4 release.


feat: improve dependency structure specification

Summary
Related to #7422
This job is a bit large, and I have temporarily managed dependencies that already have loc. I need to evaluate whether the current approach is acceptable.

Checklist

[ ] Tests updated (or not required).
[ ] Documentation updated (or not required).

The DependencyLocation in webpack is as follows. At present, I have only considered RealDependencyLocation because I have not encountered the situation of SyntheticDependencyLocation yet, and it seems that this can be considered to be ignored.
/**
 * @typedef {object} RealDependencyLocation
 * @property {SourcePosition} start
 * @property {SourcePosition=} end
 * @property {number=} index
 */

/**
 * @typedef {object} SyntheticDependencyLocation
 * @property {string} name
 * @property {number=} index
 */

/** @typedef {SyntheticDependencyLocation | RealDependencyLocation} DependencyLocation */


"ÿþ" (BOM?) in Bookmarks

I don't seem to be able to get bookmarks to work correctly.
I see #26 references encoding issues with bookmarks, but I'm not positive this
is related.
$facade = PHPPdf\Core\FacadeBuilder::create()->build();

$doc = '<pdf><page>
A

B
</page></pdf>';

$content = $facade->render($doc);

file_put_contents('./out.pdf', $content);

results in the following bookmarks:
ÿþa
ÿþb

And what I believe are the relevant lines in the PDF:
8 0 obj
<</Title (ÿþa

I can fix that behavior by changing the encoding used for rendering with the following patch. But I don't know if it's a good idea or not.
I don't know if there is a need to encode in UTF-16.
Ping @psliwa
encoding.txt

Doing what I've done in the patch breaks other things like accentuated characters like é or ë. So that's not a good idea !


Disable ivtests that fail on other simulators

Another round of disabling ivtests that fail on at least two other simulators or have other issues

br961 - Circular definition
br_gh104a - Null object dereferenced
br_gh127c - assigning to input
br_gh127f - assigning to input
br_gh230_iv - negative tests
br_gh60a - Negative test that in some tools throws a warning rather than fatal error
br_ml20180227 - negative test wrong, what is tested appears legal on other simulators and by IEEE
br_ml20181012d - Negative test in 1995/2001 but legal in SV (sv-test driver limitation)
comment1 - negative test wrong, what is tested appears legal on other simulators and by IEEE
concat4 - Checked values FAILED on multiple other simulators
enum_test3 - negative test wrong, what is tested appears legal on other simulators and by IEEE
enum_test5 - Negative test that in some tools throws a warning rather than fatal error
escaped_macro_name - Fails on others; does not appear to match IEEE's description of define identifiers
func_init_var3 -  Checked values FAILED on multiple other simulators
pr1938138 - Negative test that in some tools throws a warning rather than fatal error
pr2794144 - negative test wrong, what is tested appears legal on other simulators and by IEEE
pr2913927_iv - Illegal, f_ffffffff has too many digits for int
sv_class4_iv - Illegal, f_ffffffff has too many digits for int
vlg_pr1587634_iv - Specified --top-module 'top' was not found in design (sv-test driver limitation)
vlg_pr509_iv - Wire inputs its own output, creating circular logic (wire x=x)

Also disable hdlconvertor tests that fail on other simulators

p504_2.sv - MyBus not declared
p522.sv - Slist not declared (vs SList)
p621.sv - $async$and$array not formally part of IEEE
p622.sv - $async$and$array not formally part of IEEE

@tgorochowik et al could you consider this? thanks.

Many of these appear to be very suspect. I didn't go through all of them but for example:
br961 - All three major simulators support this just fine. The point of declaration for the port is considered to be at the name itself, not prior to it during the data type.
br_gh127c - "assigning to input" is a little too simplistic. Unfortunately the LRM allows this for net ports, in the section on "Port coercion", and all three major simulators will follow suit.
comment1 - The LRM says this in the section on comments:

Block comments shall not be nested.

br_ml20181012d - You can't have an array of width zero, what would that even mean? LRM says this:

Each fixed-size dimension shall be represented by an address range, such as [1:1024], or a single positive
number to specify the size of a fixed-size unpacked array, as in C.
This also fails on e.g. VCS.

I would also say that it seems pretty valuable to have test coverage of cases that the LRM calls out as illegal, even if by default your tool treats it as a warning instead. Surely there's a way for you to promote warnings to errors such that you could still pass the test suite? Or if you don't have any warning at all for something the LRM says is illegal, that seems like a deficiency worth rectifying in your tool. Just throwing the tests away instead seems wrong.

By nested comments isn't legal they mean you can't do /* /* */  */ and the whole thing is commented out.  Please try the test on other simulators, I did.
As to "reg x[0]" while it might seem clear that is nonsensical, I suspect you'd agree this is also IEEE illegal under the same rule:
module test();
  parameter X = 0;
  reg illegal[X];
endmodule

After elaboration this reduces internally in my code to the same case the tests cover (reg illegal[0]).  While strange, I've seen this in the "real world" where "illegal" is never used (maybe X=0 disables some feature).  This is an error on only 1 of the big 3, and so I elected not to consider it a fatal error.   (FWIW C discovered years ago that having an array of zero size is useful, SystemVerilog hasn't caught up.)
I can't promote every warning to error, nor do I wish to disable some errors (like assigning to input which I agree is technically legal but almost always "wrong"), doing such would cause false failures or false passes.  For example there's a boatload of other lint problems all through the tests (which is expected and reasonable given the tests' purposes).  You are right I could promote certain warning enables just to match what this these tests expect, but realistically that doesn't help users in any way on real designs, so I'll likely instead just keep ignoring the test failures as I have been doing previously.
Anyhow I'd greatly appreciate a review on what ones you think are reasonable to disable or not.

Here are the ones I think are pretty uncontroversial:

br_gh104a
comment1 -- After looking at this again I agree with you, my interpretation of the rule was incorrect.
concat4
escaped_macro_name
func_init_var3
vlg_pr1587634_iv

Also the hdlconvert test disables seem fine. They make no attempt to elaborate correctly, and it's not strictly clear where the line is between just parsing and full elaboration (like does name resolution count or not). I'm curious though why you say the PLA tasks aren't part of IEEE -- there's a whole section on them in the LRM?
I can do another pass on the remaining ones in your list later today or tomorrow.

I am okay with merging the changes you guys agree on.
@MikePopoloski please let us know if you intend to do the another pass you mention - if not maybe we can selectively merge the ones that are not controversial and keep the rest for discussion?

 I'll come back to this soon.

OK, here's the full list.
Uncontroversial (remove test)

br_gh104a
comment1
concat4
escaped_macro_name
func_init_var3
vlg_pr1587634_iv

After some more thought (remove test)

br_gh230_iv - some tools error on this but it's not exactly clear in the LRM what the requirements are, and some tools just warn, so it seems fine to remove this one
br_gh60a - zero sized constant is clearly illegal (LRM says must be a positive number) but bizarrely tools have chosen to warn and ignore it so I guess we should remove this one too
br_ml20180227 - illegal, string type is not implicitly assignment compatible to a packed vector, but some tools (VCS) allow it anyway so I guess we should remove
enum_test3 - definitely illegal, LRM says "Any enumeration encoding value that is outside the representable range of the enum base type shall be an error" but some tools allow it anyway
pr1938138 - illegal but some tools just warn instead
pr2794144 - unclear, most tools hard error but for some reason VCS allows it

Legal (keep test)

br961 - not a circular reference
br_gh127c - assigning to input nets is legal
br_gh127f - same as above
pr2913927_iv - Having too many vector digits is not illegal. LRM says "If the size of the unsigned number is larger than the size specified for the literal constant, the unsigned number shall be truncated from the left." All major tools support this.
sv_class4_iv - same as above
vlg_pr509_iv - Assigning a wire to itself may be pointless but it's not illegal. All major tools support this.

Illegal (keep test)

br_ml20181012d - not legal in SV, all major tools throw an error
enum_test5 - LRM is clear, "If the integer value expression is a sized literal constant, it shall be an error if the size is different from the enum base type, even if the value is within the representable range." VCS and ModelSim both error on this.

Also would still like to hear about the PLA tasks that I asked about in my earlier message.

Thanks for the review, I updated the pull request to match the ones you indicated.
You're right the PLA is still formally part of IEEE, so left those in.
I still think that allowing overflows of constants e.g. "2'd1234" is cruel to users as it's allowing something that can only be a mistake, but I'll leave the tests matching what the LRM says, and I will ask IEEE to allow tools to consider this illegal in the next LRM.

To be clear, I think it's totally reasonable for a tool to default to an error for this case, but it seems like philosophically you want a way to disable such errors when the LRM says such a construct is legal, because surely there are large codebases out there that do this and perhaps even work correctly even though it's misleading and ugly, and it's fair for sv-tests to test that tools can be compliant with such behaviors.
Also the diff lgtm but I don't have merge privileges so you'll have to wait for @tgorochowik

Thank you both @wsnyder @MikePopoloski ! I appreciate the changes and the discussion


Integration Test - Release - 2020/11/18 18:11:41

Auto create for integration test. IssueCommandFlow

@PRMergerTest2 : Thanks for your contribution! The author(s) have been notified to review your proposed change.