[FEA] Spill performance with compression enabled

I tested the worst case for spill by changing our plugin so that it would spill every buffer: https://github.com/abellina/spark-rapids/commit/84e18f156a3478139164f5630ede4327f9655732 with this patch https://github.com/NVIDIA/spark-rapids/pull/9454. I then made sure that every buffer was spilled from device to disk, forcing compression and decompression to happen. Note that we don't run with unspill on by default, so every time a buffer is read we are reading it from disk compressed, and need to decompress.
I ran NDS @ 3TB in our performance cluster in this mode. Our performance cluster has really great bandwidth to disk and lots of host memory. In this case, the compression-enabled case was 2x slower than the case where compression was not enabled, and I would catalog this as a the worst case scenario both for the spill simulated here and the fact that anything we do adds overhead in this environment, given the fast IO.

We may observe different results in slower IO or restricted host memory scenarios (a cloud VM would be good to try), where performance may be up to par with compression or even beat the uncompressed case. We should find examples where compression can be a benefit in performance and use this data to modify the auto tuner for thresholds it could use, likely relative to disk bandwidth, overall capacity, and amount of spill detected in application history logs.

We should also further study the results in our performance cluster and see if we can speed up spill. For example, right now spill is done completely serially, one buffer at a time, but we know that LZ4's bandwidth for a single thread is a limiting factor (re: multi-threaded shuffle). We could employ similar mechanisms at spill time to speed up compression (and also encryption) when we have several blocks to spill. A related issue is: https://github.com/NVIDIA/spark-rapids/issues/7666, because if you look at the stack traces that's really holding everything up (all tasks end up waiting for the catalog lock, because one task is spilling). So there are steps we can take, and we can re-run the spill-all case to figure out if we improve our worst case.

Name = benchmark
Means = 3802000.0, 7187000.0
Time diff = -3385000.0
Speedup = 0.5290107137887853
T-Test (test statistic, p value, df) = -27.525783960660327, 0.02311795793190776, 1.0
T-Test Confidence Interval = -4947553.244415962, -1822446.7555840379
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed

In terms of the amount spilled, with compression we spilled 12TB, without compression we spilled 25TB.

Full results
Name = query1
Means = 6415.0, 12243.0
Time diff = -5828.0
Speedup = 0.5239728824634485
T-Test (test statistic, p value, df) = -46.733296789404704, 0.013620323808451023, 1.0
T-Test Confidence Interval = -7412.561036590835, -4243.438963409165
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query3
Means = 8656.0, 12730.0
Time diff = -4074.0
Speedup = 0.6799685781618224
T-Test (test statistic, p value, df) = -196.01041638987795, 0.0032478592763304165, 1.0
T-Test Confidence Interval = -4338.093506098472, -3809.9064939015275
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query4
Means = 153913.0, 266906.5
Time diff = -112993.5
Speedup = 0.5766551208007299
T-Test (test statistic, p value, df) = -39.287460187701285, 0.01620064873435169, 1.0
T-Test Confidence Interval = -149537.43890637613, -76449.56109362387
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query6
Means = 2076.0, 3271.5
Time diff = -1195.5
Speedup = 0.634571297569922
T-Test (test statistic, p value, df) = -14.530994669814685, 0.04374219577439338, 1.0
T-Test Confidence Interval = -2240.870128306454, -150.12987169354642
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query7
Means = 34275.0, 75803.0
Time diff = -41528.0
Speedup = 0.4521588855322349
T-Test (test statistic, p value, df) = -41.196223331454945, 0.015450318657754002, 1.0
T-Test Confidence Interval = -54336.53504577591, -28719.464954224088
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query9
Means = 53082.0, 136078.5
Time diff = -82996.5
Speedup = 0.39008366494339664
T-Test (test statistic, p value, df) = -12.810600619381573, 0.04959419426750873, 1.0
T-Test Confidence Interval = -165316.64663011135, -676.3533698886458
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query11
Means = 78715.0, 144497.0
Time diff = -65782.0
Speedup = 0.5447517941548959
T-Test (test statistic, p value, df) = -19.47654123478562, 0.032657812876891255, 1.0
T-Test Confidence Interval = -108697.19474100177, -22866.805258998225
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query12
Means = 964.0, 1517.0
Time diff = -553.0
Speedup = 0.6354647330257086
T-Test (test statistic, p value, df) = -31.9274698861863, 0.019933045789918242, 1.0
T-Test Confidence Interval = -773.077921748727, -332.9220782512729
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query14_part1
Means = 115181.0, 221352.0
Time diff = -106171.0
Speedup = 0.5203521992121146
T-Test (test statistic, p value, df) = -18.30879791819945, 0.034736734446920055, 1.0
T-Test Confidence Interval = -179853.08820147382, -32488.911798526184
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query14_part2
Means = 109677.0, 206264.0
Time diff = -96587.0
Speedup = 0.5317311794593337
T-Test (test statistic, p value, df) = -21.01942346408533, 0.030264394239590434, 1.0
T-Test Confidence Interval = -154973.6726399373, -38200.327360062714
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query16
Means = 52068.0, 148311.0
Time diff = -96243.0
Speedup = 0.3510730829136072
T-Test (test statistic, p value, df) = -22.173153215330068, 0.02869184503407946, 1.0
T-Test Confidence Interval = -151394.527190231, -41091.472809769
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query17
Means = 7584.0, 13227.0
Time diff = -5643.0
Speedup = 0.5733726468586982
T-Test (test statistic, p value, df) = -814.4968922592645, 0.0007816106587309643, 1.0
T-Test Confidence Interval = -5731.031168699491, -5554.968831300509
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query22
Means = 8517.0, 16893.0
Time diff = -8376.0
Speedup = 0.5041733262297994
T-Test (test statistic, p value, df) = -19.038920687922463, 0.033407109622622284, 1.0
T-Test Confidence Interval = -13965.979212417667, -2786.0207875823335
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query23_part1
Means = 209113.0, 519179.5
Time diff = -310066.5
Speedup = 0.4027759185406974
T-Test (test statistic, p value, df) = -206.35962794430557, 0.0030849774036093864, 1.0
T-Test Confidence Interval = -329158.25971170206, -290974.74028829794
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query24_part1
Means = 113715.0, 196010.5
Time diff = -82295.5
Speedup = 0.5801474920986376
T-Test (test statistic, p value, df) = -20.608687520318735, 0.030866635017350957, 1.0
T-Test Confidence Interval = -133034.46485916903, -31556.53514083098
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query24_part2
Means = 117606.0, 198203.5
Time diff = -80597.5
Speedup = 0.5933598548966088
T-Test (test statistic, p value, df) = -24.44601435303959, 0.02602735258808861, 1.0
T-Test Confidence Interval = -122489.33240487019, -38705.6675951298
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query25
Means = 6139.0, 9359.5
Time diff = -3220.5
Speedup = 0.6559111063625194
T-Test (test statistic, p value, df) = -19.07032350692502, 0.03335219931563161, 1.0
T-Test Confidence Interval = -5366.259737050089, -1074.740262949911
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query26
Means = 16413.0, 40341.0
Time diff = -23928.0
Speedup = 0.4068565479289061
T-Test (test statistic, p value, df) = -23.179257116055982, 0.02744804303055482, 1.0
T-Test Confidence Interval = -37044.64413622413, -10811.355863775869
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query29
Means = 13873.0, 23951.0
Time diff = -10078.0
Speedup = 0.5792242495094151
T-Test (test statistic, p value, df) = -138.53657173554876, 0.004595239422516207, 1.0
T-Test Confidence Interval = -11002.327271344653, -9153.672728655347
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query31
Means = 14874.0, 21506.0
Time diff = -6632.0
Speedup = 0.6916209429926532
T-Test (test statistic, p value, df) = -46.132373316452984, 0.0137976878869646, 1.0
T-Test Confidence Interval = -8458.646750514436, -4805.353249485565
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query33
Means = 3552.0, 5333.0
Time diff = -1781.0
Speedup = 0.6660416276017251
T-Test (test statistic, p value, df) = -20.984914886259663, 0.030314087294278914, 1.0
T-Test Confidence Interval = -2859.3818165687626, -702.6181834312374
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query35
Means = 9942.0, 13643.5
Time diff = -3701.5
Speedup = 0.7286986477076997
T-Test (test statistic, p value, df) = -83.80635378060391, 0.007595958211485061, 1.0
T-Test Confidence Interval = -4262.698700459254, -3140.301299540746
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query36
Means = 20697.0, 37183.0
Time diff = -16486.0
Speedup = 0.5566253395368851
T-Test (test statistic, p value, df) = -85.74951835910063, 0.0074238424536921714, 1.0
T-Test Confidence Interval = -18928.86493141087, -14043.13506858913
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query38
Means = 27160.0, 43873.0
Time diff = -16713.0
Speedup = 0.6190595582704624
T-Test (test statistic, p value, df) = -16.162906279675404, 0.039337561907621145, 1.0
T-Test Confidence Interval = -29851.651928399006, -3574.3480716009963
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query40
Means = 4458.0, 7929.0
Time diff = -3471.0
Speedup = 0.5622398789254635
T-Test (test statistic, p value, df) = -250.4978480446489, 0.0025414046290069396, 1.0
T-Test Confidence Interval = -3647.0623373989815, -3294.9376626010185
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query42
Means = 2535.0, 3865.0
Time diff = -1330.0
Speedup = 0.6558861578266494
T-Test (test statistic, p value, df) = -127.97930967036704, 0.0049742948155542515, 1.0
T-Test Confidence Interval = -1462.0467530492363, -1197.9532469507637
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query43
Means = 15179.0, 25691.0
Time diff = -10512.0
Speedup = 0.5908294733564283
T-Test (test statistic, p value, df) = -13.079969891640832, 0.04857685119061344, 1.0
T-Test Confidence Interval = -20723.615569140937, -300.3844308590651
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query45
Means = 3242.0, 5554.0
Time diff = -2312.0
Speedup = 0.583723442563918
T-Test (test statistic, p value, df) = -34.226508265805506, 0.01859490638348658, 1.0
T-Test Confidence Interval = -3170.3038948200356, -1453.6961051799644
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query49
Means = 5589.0, 8519.0
Time diff = -2930.0
Speedup = 0.6560629181828853
T-Test (test statistic, p value, df) = -187.95958763617816, 0.003386971496688935, 1.0
T-Test Confidence Interval = -3128.0701295738545, -2731.9298704261455
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query50
Means = 40862.0, 54916.5
Time diff = -14054.5
Speedup = 0.7440750958273015
T-Test (test statistic, p value, df) = -25.719078790255452, 0.02474035940893158, 1.0
T-Test Confidence Interval = -20997.958431172337, -7111.041568827662
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query53
Means = 18280.0, 29691.5
Time diff = -11411.5
Speedup = 0.615664415741879
T-Test (test statistic, p value, df) = -16.207706265331893, 0.0392291037318845, 1.0
T-Test Confidence Interval = -20357.667519085753, -2465.332480914247
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query59
Means = 54282.0, 82104.0
Time diff = -27822.0
Speedup = 0.6611370944168372
T-Test (test statistic, p value, df) = -180.48358639768276, 0.00352726400587758, 1.0
T-Test Confidence Interval = -29780.69350356367, -25863.30649643633
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query61
Means = 5088.0, 8353.0
Time diff = -3265.0
Speedup = 0.6091224709685144
T-Test (test statistic, p value, df) = -117.81553930650801, 0.005403399998578176, 1.0
T-Test Confidence Interval = -3617.124674797963, -2912.875325202037
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query62
Means = 65220.0, 135827.5
Time diff = -70607.5
Speedup = 0.48016785996944655
T-Test (test statistic, p value, df) = -26.600495355175532, 0.023921363714386055, 1.0
T-Test Confidence Interval = -104334.44150799242, -36880.558492007585
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query64
Means = 115962.0, 262443.5
Time diff = -146481.5
Speedup = 0.4418551040509672
T-Test (test statistic, p value, df) = -25.121382283172483, 0.025328376725672595, 1.0
T-Test Confidence Interval = -220570.73235670896, -72392.26764329104
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query65
Means = 35580.0, 82682.0
Time diff = -47102.0
Speedup = 0.4303234077550132
T-Test (test statistic, p value, df) = -51.60218667812097, 0.012335525640321495, 1.0
T-Test Confidence Interval = -58700.10647615792, -35503.89352384208
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query66
Means = 41169.0, 67463.0
Time diff = -26294.0
Speedup = 0.6102456161155003
T-Test (test statistic, p value, df) = -15.459111993963361, 0.04112358088199367, 1.0
T-Test Confidence Interval = -47905.65191572499, -4682.348084275007
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query69
Means = 5726.0, 7168.0
Time diff = -1442.0
Speedup = 0.798828125
T-Test (test statistic, p value, df) = -277.5130293904801, 0.0022940076663786795, 1.0
T-Test Confidence Interval = -1508.023376524618, -1375.976623475382
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query71
Means = 7515.0, 10410.0
Time diff = -2895.0
Speedup = 0.7219020172910663
T-Test (test statistic, p value, df) = -417.85725732599167, 0.0015235311720830304, 1.0
T-Test Confidence Interval = -2983.031168699491, -2806.968831300509
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query72
Means = 35595.0, 64122.0
Time diff = -28527.0
Speedup = 0.5551136895293347
T-Test (test statistic, p value, df) = -43.803380662692696, 0.01453105217548824, 1.0
T-Test Confidence Interval = -36801.92985775214, -20252.070142247863
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query74
Means = 51937.0, 87044.5
Time diff = -35107.5
Speedup = 0.596671817288858
T-Test (test statistic, p value, df) = -16.580224601697168, 0.03834987389173396, 1.0
T-Test Confidence Interval = -62012.02593378188, -8202.974066218121
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query75
Means = 69209.0, 130502.5
Time diff = -61293.5
Speedup = 0.5303270052297848
T-Test (test statistic, p value, df) = -27.85345826412777, 0.022846227941127655, 1.0
T-Test Confidence Interval = -89254.39995817577, -33332.60004182423
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query78
Means = 121943.0, 238542.0
Time diff = -116599.0
Speedup = 0.5112013817273269
T-Test (test statistic, p value, df) = -23.645403595799497, 0.026907581835764048, 1.0
T-Test Confidence Interval = -179255.1843218626, -53942.815678137405
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query79
Means = 18694.0, 31362.5
Time diff = -12668.5
Speedup = 0.5960621761658031
T-Test (test statistic, p value, df) = -25.708829122069506, 0.02475021303093756, 1.0
T-Test Confidence Interval = -18929.716873751284, -6407.283126248716
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query80
Means = 16541.0, 33240.5
Time diff = -16699.5
Speedup = 0.4976158601705751
T-Test (test statistic, p value, df) = -31.559609886520967, 0.020165231636396767, 1.0
T-Test Confidence Interval = -23422.880509423612, -9976.119490576388
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query82
Means = 12153.0, 24937.5
Time diff = -12784.5
Speedup = 0.4873383458646617
T-Test (test statistic, p value, df) = -49.04408316581243, 0.012978763787733116, 1.0
T-Test Confidence Interval = -16096.672722318342, -9472.327277681658
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query86
Means = 7621.0, 11501.0
Time diff = -3880.0
Speedup = 0.6626380314755239
T-Test (test statistic, p value, df) = -21.133198532601398, 0.030101702837526164, 1.0
T-Test Confidence Interval = -6212.825970536507, -1547.1740294634933
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query87
Means = 27979.0, 46990.5
Time diff = -19011.5
Speedup = 0.5954182228322746
T-Test (test statistic, p value, df) = -21.128574865637287, 0.030108280319781196, 1.0
T-Test Confidence Interval = -30444.548034846368, -7578.45196515363
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query88
Means = 252115.0, 367616.5
Time diff = -115501.5
Speedup = 0.685809804510951
T-Test (test statistic, p value, df) = -26.818750097247364, 0.023726869273490603, 1.0
T-Test Confidence Interval = -170223.87524282097, -60779.12475717902
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query90
Means = 15768.0, 33103.5
Time diff = -17335.5
Speedup = 0.4763242557433504
T-Test (test statistic, p value, df) = -29.832058395042495, 0.021332134620458392, 1.0
T-Test Confidence Interval = -24719.11427466979, -9951.885725330209
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query91
Means = 1618.0, 2119.0
Time diff = -501.0
Speedup = 0.7635677206229353
T-Test (test statistic, p value, df) = -28.92524848640025, 0.022000375278840318, 1.0
T-Test Confidence Interval = -721.077921748727, -280.9220782512729
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query93
Means = 50638.0, 102276.5
Time diff = -51638.5
Speedup = 0.49510884709586267
T-Test (test statistic, p value, df) = -46.04401834061543, 0.013824156288149609, 1.0
T-Test Confidence Interval = -65888.54543323007, -37388.45456676993
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query95
Means = 209807.0, 354026.0
Time diff = -144219.0
Speedup = 0.5926316146271743
T-Test (test statistic, p value, df) = -24.685703668028058, 0.025774913774320417, 1.0
T-Test Confidence Interval = -218451.28300584562, -69986.71699415437
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query97
Means = 23448.0, 29446.0
Time diff = -5998.0
Speedup = 0.796305100862596
T-Test (test statistic, p value, df) = -15.59885997567286, 0.04075617088568512, 1.0
T-Test Confidence Interval = -10883.72986282174, -1112.2701371782596
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query98
Means = 3617.0, 4688.5
Time diff = -1071.5
Speedup = 0.7714620880878745
T-Test (test statistic, p value, df) = -16.94878940922422, 0.037517876548842255, 1.0
T-Test Confidence Interval = -1874.7844143828538, -268.2155856171463
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = query99
Means = 122054.0, 239049.5
Time diff = -116995.5
Speedup = 0.5105804446359437
T-Test (test statistic, p value, df) = -57.3164051073185, 0.011105985933025706, 1.0
T-Test Confidence Interval = -142931.6830780875, -91059.31692191251
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed
--------------------------------------------------------------------
Name = benchmark
Means = 3802000.0, 7187000.0
Time diff = -3385000.0
Speedup = 0.5290107137887853
T-Test (test statistic, p value, df) = -27.525783960660327, 0.02311795793190776, 1.0
T-Test Confidence Interval = -4947553.244415962, -1822446.7555840379
ALERT: significant change has been detected (p-value < 0.05)
ALERT: regression in performance has been observed

Configuration used
No compression:
export SPARK_CONF=("--master" "spark://master-node:7077"
                   "--conf" "spark.shuffle.spill.compress=false"
                   "--conf" "spark.rapids.memory.host.spillStorageSize=1"
                   "--conf" "spark.locality.wait=0"
                   "--conf" "spark.plugins=com.nvidia.spark.SQLPlugin"
                   "--conf" "spark.sql.adaptive.enabled=true"
                   "--conf" "spark.sql.files.maxPartitionBytes=2gb"
                   "--conf" "spark.driver.maxResultSize=2GB"
                   "--conf" "spark.driver.memory=50G"
                   "--conf" "spark.executor.cores=16"
                   "--conf" "spark.executor.memory=16G"
                   "--conf" "spark.executor.resource.gpu.amount=1"
                   "--conf" "spark.task.resource.gpu.amount=0.0625"
                   "--conf" "spark.rapids.memory.pinnedPool.size=8g"
                   "--conf" "spark.rapids.sql.concurrentGpuTasks=4"
                   "--conf" "spark.executor.extraJavaOptions=-Dai.rapids.cudf.nvtx.enabled=true"
                   "--conf" "spark.driver.extraClassPath=$SPARK_RAPIDS_PLUGIN_JAR"
                   "--conf" "spark.executor.extraClassPath=$SPARK_RAPIDS_PLUGIN_JAR"
                   "--conf" "spark.shuffle.manager=com.nvidia.spark.rapids.spark321.RapidsShuffleManager"
                   "--conf" "spark.rapids.shuffle.multiThreaded.writer.threads=32"
                   "--conf" "spark.rapids.shuffle.multiThreaded.reader.threads=32"
                   "--conf" "spark.rapids.shuffle.mode=MULTITHREADED")

Compression:
export SPARK_CONF=("--master" "spark://master-node:7077"
                   "--conf" "spark.shuffle.spill.compress=true"
                   "--conf" "spark.rapids.memory.host.spillStorageSize=1"
                   "--conf" "spark.locality.wait=0"
                   "--conf" "spark.plugins=com.nvidia.spark.SQLPlugin"
                   "--conf" "spark.sql.adaptive.enabled=true"
                   "--conf" "spark.sql.files.maxPartitionBytes=2gb"
                   "--conf" "spark.driver.maxResultSize=2GB"
                   "--conf" "spark.driver.memory=50G"
                   "--conf" "spark.executor.cores=16"
                   "--conf" "spark.executor.memory=16G"
                   "--conf" "spark.executor.resource.gpu.amount=1"
                   "--conf" "spark.task.resource.gpu.amount=0.0625"
                   "--conf" "spark.rapids.memory.pinnedPool.size=8g"
                   "--conf" "spark.rapids.sql.concurrentGpuTasks=4"
                   "--conf" "spark.executor.extraJavaOptions=-Dai.rapids.cudf.nvtx.enabled=true"
                   "--conf" "spark.driver.extraClassPath=$SPARK_RAPIDS_PLUGIN_JAR"
                   "--conf" "spark.executor.extraClassPath=$SPARK_RAPIDS_PLUGIN_JAR"
                   "--conf" "spark.shuffle.manager=com.nvidia.spark.rapids.spark321.RapidsShuffleManager"
                   "--conf" "spark.rapids.shuffle.multiThreaded.writer.threads=32"
                   "--conf" "spark.rapids.shuffle.multiThreaded.reader.threads=32"
                   "--conf" "spark.rapids.shuffle.mode=MULTITHREADED")

This work can help inform this autotuner enhancement: https://github.com/NVIDIA/spark-rapids-tools/issues/644.


ValueError: could not broadcast input array from shape (2,15) into shape (2,16)

When I ran train.py, this bug appeared. Later, I adjusted the torch version and no longer reported the error, but it appeared again when I ran evaluate_model.py. I don't know what caused it?

@liusui652 I'm having same problem when I run train.py. What torch version did you adjusted?

Hi @liusui652 ! Did you yolved the problem? I'm currently facing the same issue

It is because some person is not in all 16 positions (8 observed, 8 predicted) - in your case,  "(2,15)" means there are 15 positions of some person in dataset instead of 16. I had the same issue so I added these lines in trajectory.py under line curr_ped_seq = np.transpose(curr_ped_seq[:, 2:]):
if curr_ped_seq.shape != (2,16): continue
(in case you are predicting 12 positions, use (2,20) )


Make Virtio transmit queued packets

The packets can allready chain together, (e.g. used by Arp, when a packet is waiting for resolution). Make the device actually transmit all the packets in the chain.

Fixed after 7d8f55c06d2d43d49a50c01cb3878bf375d3c25e


Prevent touch when clicking on the background of the loading widget

What it should normally be, background widgets should not be clickable when the loading widget is open. So what I want is to block the click of a TextField, Button, etc on the background of the loading widget.

This is already developed feature, not an issue.
  void configLoadingIndicator() {
    EasyLoading.instance
      ..userInteractions = false
  }

then call configLoadingIndicator() in main.dart.

@berkanaslan Yes you are right! Thank you for your help.


Basics of MySQL

Kindly review.

@happycoder0011 Kindly edit your PR description and follow this template for the same in case of new MicroBytes.
Also, your PR is under review.

@happycoder0011 Pls work on my comments. We are eager to see ur completed contribution. U will get  creator badges only if u do the needed latest by today midnight

@happycoder0011 Pls work on my comments. We are eager to see ur completed contribution. U will get creator badges only if u do the needed latest by today midnight

I have made the changes. Kindly, review.

@happycoder0011
Sorry for the delay, kindly edit your PR description and follow this template for the same in case of new MicroBytes.
Also, your PR is under review.

Kindly do this ASAP!

Adding to these, a couple more:

In second terminal image of describe student command mention there is an alternative way as heading to diff the photos.
Use proper formatting of bullets. Like after a heading like Insert values in table  use bullets for the two subsequent cases. Do this for all such cases. Also, use a colon(:) after the heading separating it from its description.
Before ur tasks mention the heading as a challenge, then sub-headings as tasks.
Don't mention extra bytes, call it anything else like trivia/test urself/etc. What u r creating is a microbyte. Within that don't mention "bytes"
Center align all tables

Kindly do these ASAP

We have given you the label seeing the potential in ur PR and what we have achieved together. Kindly wrap-up ur microbyte with respect to our last comments. We are making an exception for you; u can complete this by maximum tomorrow night. After that we are sorry but we will have to close this PR which we don't want to since this is quality work. Kindly do the necessary ASAP and u can still get the creator badges and be considered for the other perks!

Adding to these, a couple more:

In second terminal image of describe student command mention there is an alternative way as heading to diff the photos.
Use proper formatting of bullets. Like after a heading like Insert values in table  use bullets for the two subsequent cases. Do this for all such cases. Also, use a colon(:) after the heading separating it from its description.
Before ur tasks mention the heading as a challenge, then sub-headings as tasks.
Don't mention extra bytes, call it anything else like trivia/test urself/etc. What u r creating is a microbyte. Within that don't mention "bytes"
Center align all tables

Kindly do these ASAP

I have made all the changes you noted. How to center the table?

Great work @happycoder0011 . Happy we could guide u. Another advocate will be shortly reviewing ur PR one last time and after their approval, we will merge it. Two things to do after merging only:

Open as many learners issues as u can(within the scope of ur microbyte)
Showcase ur newly merged microbyte in the #mirobyte-showcase channel in slack using this template

Also, fill this form right away.
Happy Contributing!!

Thanks a lot for your efforts, patience and guidance throughout the process.

Hey @happycoder0011!
Thanks for working with us by making timely changes even after the challenge.
I appreciate all the effort you have put in by adding tables, images etc.
This MicroByte can be used a "getting started" guide for needy learners.
Also thanks for reverting the changes to CONTRIBUTING.md. It would make it hard to manage without that.
Cheers 😄

Thanks a lot!!

@all-contributors kindly add @happycoder0011 for:

[x] code
[x] content
[x] ideas
[x] doc


fix(service overview): remove multiple errors

Description
Service overview: Added missing translations and fixed duplicate error
Package: Updated shared components dependency and removed duplicate log
Why
Service overview: since only one error should be displayed
Issue
Ref: CPLP-3545, CPLP-3473
Checklist
Please delete options that are not relevant.

[x] I have followed the contributing guidelines
[x] I have performed a self-review of my own code
[x] I have successfully tested my changes locally

@lavanya-bmw as mentioned in this pr #404 the changeLog was incorrectly updated. Now looking at this I can still see here the incorrect changeLog as well as on dev. Was this not fixed with Evelyn jointly? This needs to get adjusted before we can move ahead. Changing a previous ChangeLog is a major issue...also please make sure that review comments are completely handled/managed.

@jjeroch I have reverted the changelog from previous release


If a scenario has an image, it is getting passed to all questions in the survey; need way to indicate only goes to some questions

The problem is that the API call for a model includes an encoded image if that image is present in the scenario. As such, the image is send in the prompt to all questions in the Survey---not just the intended one.
Possible solutions:

Require the prompt to explicitly mention image via {{ image }}
Include an "image = True" parameter in the question construction

I think 1 is better because it will make it easier later to turn these into human-readable surveys.

For an image scenario, change the repr to only show the first bit


change of path in calico Install-Containerd script

The Install-Containerd script provided by calico recently moved the script to a different URL. This PR updates the download location.
Otherway around will be keeping a local copy of the script to avoid any issue in future.

/approve

/lgtm


bug: wrong value for ks_tag in v0.9.0 mkdocs.yml

Describe the bug
The bootstrap commands displayed in https://docs.kubestellar.io/release-0.9/Getting-Started/quickstart/#2-install-and-run-kcp-and-kubestellar say --kubestellar-version latest but they should say --kubestellar-version v0.9.0.
The cause of the problem is at https://github.com/kubestellar/kubestellar/blob/v0.9.0/docs/mkdocs.yml#L11 . Older releases specify their own point release semver there.
Steps To Reproduce
See above
Expected Behavior
See above
Additional Context
No response

fixed - I need to update the release instructions to cover this field.


[pytorch] Fix pytorch engine version string

Change-Id: I87a40e4f30e427b8ec72b6fa09f4784535dca0c2
Description
Brief description of what this PR is about

If this change is a backward incompatible change, why must this change be made?
Interesting edge cases to note here

Codecov Report

Merging #1469 (97be5a5) into master (bb5073f) will decrease coverage by 1.43%.
The diff coverage is 48.12%.

@@             Coverage Diff              @@
##             master    #1469      +/-   ##
============================================
- Coverage     72.08%   70.65%   -1.44%
- Complexity     5126     5299     +173
============================================
  Files           473      493      +20
  Lines         21970    23227    +1257
  Branches       2351     2535     +184
============================================
+ Hits          15838    16412     +574
- Misses         4925     5537     +612
- Partials       1207     1278      +71

Impacted Files
Coverage Δ

api/src/main/java/ai/djl/modality/cv/Image.java
69.23% <ø> (-4.11%)
:arrow_down:

...i/djl/modality/cv/translator/BigGANTranslator.java
21.42% <ø> (-5.24%)
:arrow_down:

...odality/cv/translator/BigGANTranslatorFactory.java
33.33% <0.00%> (+8.33%)
:arrow_up:

...nslator/InstanceSegmentationTranslatorFactory.java
14.28% <0.00%> (-3.90%)
:arrow_down:

.../modality/cv/translator/YoloTranslatorFactory.java
8.33% <0.00%> (-1.67%)
:arrow_down:

...i/djl/modality/cv/translator/YoloV5Translator.java
5.69% <0.00%> (ø)

...odality/cv/translator/YoloV5TranslatorFactory.java
8.33% <0.00%> (-1.67%)
:arrow_down:

...pi/src/main/java/ai/djl/ndarray/BytesSupplier.java
54.54% <0.00%> (-12.13%)
:arrow_down:

...i/src/main/java/ai/djl/ndarray/NDArrayAdapter.java
15.23% <0.00%> (-0.83%)
:arrow_down:

...l/training/loss/SigmoidBinaryCrossEntropyLoss.java
64.00% <0.00%> (ø)

... and 162 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1fdf6fa...97be5a5. Read the comment docs.


The module '@nosto/nosto-react' has no exported member 'NostoProvider

The Build Proccess Failed to compile.
09:24:00.991 | ./pages/_app.tsx:10:10
09:24:00.991 | Type error: Module '"@nosto/nosto-react"' has no exported member 'NostoProvider'.
09:24:01.091 | error Command failed with exit code 1.
09:24:01.092 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
09:24:01.111 | Error: Command "yarn run build" exited with 1
The error is occurring because the module '@nosto/nosto-react' does not have a member named 'NostoProvider' that can be imported and used in the code.
In the given code snippet, the line import { NostoProvider } from "@nosto/nosto-react"; is trying to import the 'NostoProvider' member from the module '@nosto/nosto-react'. However, it seems that the module does not export a member with that name.

Any updates on this issue? 🙂

Hello @AndreasHegenbarth Can you please share the version of nosto-react that you are using?

Hello @AndreasHegenbarth Can you please share the version of nosto-react that you are using?

I cannot answer for @AndreasHegenbarth, but we have the same problem using nosto-react in version 0.4.3.
Here is a CodeSandbox that reproduces the issue.

@danielbigler Thanks for the CodeSanbox example.

Hello @AndreasHegenbarth @danielbigler
We have released a new version 0.5.0 addressing this issue https://www.npmjs.com/package/@nosto/nosto-react
Can you please give it a try and share your feedback?

Thank you for the fix @manikandan-ravikumar. Works fine on our side now 👍

Glad it's working for you


Add ndjson-join?

It’d be cool to join two ndjson streams (e.g., a stream from a CSV file and a stream of features from a GeoJSON collection).

Hypothetical example using bash process substitution:
ndjson-join <(shp2json -n example.shp) <(csv2json -n example.csv)

So, you’d specify an expression that takes d and i, and is evaluated for each line in each input A and B (in parallel, hopefully), and put into a Map of [a, b] for each evaluated key.
If an object in A has no corresponding key in B, then the corresponding entry is [a, null]. Likewise if an object in B has no corresponding key in A then the corresponding entry is [null, b].
But what if multiple objects in A have the same key, or multiple objects in B have the same key? I believe you’d want the cartesian product in that case, so for example if a1, a2, b1 and b2 have the same key, you should see [a1, b1], [a1, b2], [a2, b1] and [a2, b2] in the output stream.
You might also want options like whether it’s an inner join (don’t output any [a, null] or [null, b]), a left outer join (don’t output any [null, b]), a right outer join (don’t output any [a, null]), or a full outer join (include all results). Although it’d also be possible to do that in a subsequent ndjson-filter.


AUTO_INCREMENT is not supported.

It was added recently in https://github.com/ballista-compute/sqlparser-rs/pull/234
Please check out https://github.com/ballista-compute/sqlparser-rs#contributing when you get a chance.


accessibleNameBehavior should be a mutator key

While looking into the best pattern for https://github.com/phetsims/ratio-and-proportion/issues/135, I saw that accessibleNameBehavior wasn't being mutated. This may have been the bug causing weird patterns for common code accessibleName setting in https://github.com/phetsims/sun/issues/503#issuecomment-488736665 (copied over to https://github.com/phetsims/sun/issues/444. We should add this in.

Thanks @jessegreenberg for helping me track down this.


⚠ Processors has degraded performance

In 08408b2, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 90 ms

Resolved: Processors performance has improved in ca6bc24.


/G doesn't work

It worked before the update to show touches http://prntscr.com/655ozt

We need to know the exact error, please past the error

I'm working on revising the command and chat system, the issue will be addressed soon

Okey. The exact error is something simmilar to "This command doesnt exist"

Me happens the same, its says that that command doesn't exist..
El dia 14/02/2015 23.05, "Elliott"<EMAIL_ADDRESS>va escriure:

We need to know the exact error, please past the error
—
Reply to this email directly or view it on GitHub
https://github.com/twizmwazin/CardinalPGM/issues/174#issuecomment-74393969
.

Thank you! Keep doing this work. Btw, i have made a tutorial to compile the plugin in spanish. Would you consider including it on the Spanish guide?

Go ahead, add it


Improved readme and fixed crash for empty image source

Fixed crashes when uri is empty:
<CacheableImage source={{uri:''}}/>

@eugenehp I'm going to cherry pick your changes to leave out Nathan's changes.

@jayesbe sounds good. let me know if you need any help with it.

@eugenehp added your readme changes and empty source fix. I left out nathan's changes for now.. seeking a better option.

Yup already pushed 1.4.3

Awesome!
On Sat, Nov 26, 2016 at 2:33 AM, Jesse<EMAIL_ADDRESS>wrote:

Yup already pushed 1.4.3
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jayesbe/react-native-cacheable-image/pull/29#issuecomment-263056206,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABxW723NynDG0Z1hRDHC3eOq99egfyNcks5rCAr4gaJpZM4K80b3
.


fix: ensure beforeNavigate works on other navigations after clicking a download link

fixes https://github.com/sveltejs/kit/issues/9744
Clicking on a link without the download attribute, but returns the response header content-disposition: attachment, will trigger a download instead of an external navigation. Currently, Kit thinks it's an external navigation and that the page will unload.
This fix moves the before_navigate call out of the click event handler and solely into the beforeunload event handler (there was already one there but would check the navigating flag to prevent it from stacking with the one from the click event handler). Hence, only links that will actually unload the page (external links) will be treated as such, and not get mixed up with these implicit download links (they trigger a download but have no download attribute).
Notes:
Don't know if there's a way to determine if a link is a download before receiving the response. Therefore, beforeNavigate will always run for these types of implicit download links.
Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

[x] It's really useful if your PR references an issue where it is discussed ahead of time. In many cases, features are absent for a reason. For large changes, please create an RFC: https://github.com/sveltejs/rfcs
[x] This message body should clearly illustrate what problems it solves.
[x] Ideally, include a test that fails without this PR but passes with it.

Tests

[x] Run the tests with pnpm test and lint the project with pnpm lint and pnpm check

Changesets

[ ] If your PR makes a change that should be noted in one or more packages' changelogs, generate a changeset by running pnpm changeset and following the prompts. Changesets that add features should be minor and those that fix bugs should be patch. Please prefix changeset messages with feat:, fix:, or chore:.

Shoot the webkit checks are failing. Probably broke something

Cancelling beforeunload is currently broken in Safari. It only works when the page is in an iframe.

Regarding the Safari bug: Does this introduce a new bug for Safari users that wasn't there before, or does it "just" not fix it for them? In the latter case I'd be ok with not executing that test until it's fixed in Safari.

or does it "just" not fix it for them? In the latter case I'd be ok with not executing that test until it's fixed in Safari.

Yes, exactly that. On the bright side, it's been marked as "on their radar" and I'm hoping it's fixed soon.

Yes, exactly that. All websites are unable to cancel beforeunload in Safari. On the bright side, it's been marked as "on their radar" and I'm hoping it's fixed soon.

If this means that this PR fixes the problem for all but safari users and there's no way to solve it differently then I'm ok to skip this test for MacOs. I'm a bit surprised though that apparently .preventDefault() of beforeunload really does not work at all? Do we have no other tests currently that check that simply closing the browser triggers our beforeunload (which would then already have failed on Safari)?

Do we have no other tests currently that check that simply closing the browser triggers our beforeunload (which would then already have failed on Safari)?

We do have several tests. In hindsight, this changes the way links get cancelled. Currently, it cancels the click event when cancel is called in beforeNavigate. This PR changes that to cancel the navigation when the beforeunload event occurs (which is broken in Safari)

Ok so that means that it does break Safari in ways it didn't before ... shit

Presumably this would also prevent people from providing their own cancellation UI in the case where an external link is clicked (and there's unsaved work or whatever)?
I think we need to keep the existing behaviour but somehow determine more reliably whether a beforeunload event is a direct consequence of a link click. Perhaps if we checked that document.activeElement was equal to the clicked <a> (and that no focusout events had taken place in the meantime)?

After merging the latest changes from master this seems to be working just fine? Not sure what the exact change was.
Feel free to close or merge for the tests.

Unfortunately the test is flawed — because /before-navigate/download is an internal link, the failure state isn't reached. I've tried to update the test to point at an external URL (that we manage inside the test) but I can't seem to get it to run.

Ah shoot you're right. It's still broken after I fixed the test. Let me try to add a fix in again.

The Ubuntu tests are failing because Firefox is navigating to the download URL instead of just downloading. Not sure how to proceed from here. @Rich-Harris would replacing the download URL with the external one help get around this quirky behaviour?

I've no idea why the tests are so flaky even with the hacky "wait for idle network" statements. Maybe I'll take another stab at it after how to properly write playwright tests :/

Ok I think the page.waitForEvent('download') fixes that flaky download test.
Small note, the current behaviour for an implicit download link is:

Clicking on the link is treated as a link navigation.
beforeNavigate runs, but we fallback to a native navigation because the endpoint route is not a page.
The native navigation triggers and the browser receives the response with the content-disposition: attachment header.
From the header, the browser knows it's a download, so the browser doesn't navigate. We also know this in SvelteKit because we changed the location.href to the download URL when navigating natively, but afterwards, the location.href is still the same. (might be bad if it wasn't actually a download URL and it redirected us back to the same page?).
we cancel our client-side navigation and never run afterNavigate because now we know it wasn't a real navigation (not sure if this is right).


Add IsTrajectoryFrozen to PoseGraph interface

Adds IsTrajectoryFrozen to the PoseGraph interface

@wally-the-cartographer merge

Merge requested by authorized user gaschler. Merge queue now has a length of 1.

Refusing to merge, since the combined checks state is failure

@wally-the-cartographer merge

Merge requested by authorized user gaschler. Merge queue now has a length of 1.


Implementation difference of 'hash' type multiformat and 'multihash'

I have been working on making a CAR block decoder in python as a personal project. When testing HAMT-shards, I got the exception: multiformats.multihash.err.MultihashValueError: Multicodec 'murmur3-x64-64' exists, but it is not a multihash multicodec.
It seems that the type of the murmur3 was recently changed to that of a 'hash' as it is not cryptographic (https://github.com/multiformats/multihash/issues/157).
Changing lines https://github.com/hashberg-io/multiformats/blob/99afd7690ae773ddb84091b4f984c7fe279bce67/multiformats/multihash/raw.py#L122 and https://github.com/hashberg-io/multiformats/blob/99afd7690ae773ddb84091b4f984c7fe279bce67/multiformats/multihash/init.py#L59 to allow the 'hash' type allowed me to hash normally with:
>>> from multiformats import multihash
>>> mh = multihash.get(code=0x22)
>>> mh.digest(b'test')
b'"\x08\x9a\x12\x821\xf9\xbdM\x82'

Should the 'hash' type be allowed with multihashes, or should something else be done with them?

Happy with those suggested changes, they have now been committed. I hope this helps you!


browser.saveScreenshot(filepath); - Why its not accepting filePath as outside of project Directory?

Can you please provide a description of your problem?

browser.saveScreenshot(filepath);
Project directory - C:/Project
filepath - C:/Logs/Screenshots/
name - project_QA_0416202408011234
browser.saveScreenshots(filepath+name.png);
Getting error as cannot find path to savescreenshot as its concatinating the path - 'C:/ProjectC:/Logs/Screenshots/'

'C:/ProjectC:/Logs/Screenshots/'

This is not a valid path, I recommend use Node.js utilities like path.join(...). This doesn't seem like a WebdriverIO issue, therefor I will close it. I recommend to join the discord channel for questions like this.

That is what why its merging project directory path wit filepath?


Single Quote as an apostrophe in Search Text vs Smart Quote - search fails for words across the two types.

Currently, typing a single quote ' in a search field doesn't find articles where a smart quote ՚ exists, for example
For example, try Day՚s  in the title  vs Day's
The issue is smart quote/apostrophe is really:
HTML Entity: ’ or ’ or ’
The single quote you type on your keyboard is different.  The authoring apps are changing these to smart (curly) quotes.
Smart quotes also exist throughout the database surrounding single quoted phrases, thus involving a &lsquo.
This would also affect double quotes, but these are not allowed in searches anyway.
I am thinking about possible solutions to improving searches where single quotes are part of the search criteria.

After doing some research, I believe the curly apostrophe is the preferred and correct apostrophe to use in English and in French, at least.
I am thinking about/proposing the following for apostrophes:

Have the data processor stage replace act like a modern word processor and replace all ' with ՚ in the context of an apostrophe or single char quote.  Likewise for left curly single quote.  This only impacts EXP_ARCH1 (generated) files, not KBD3 files.
Have the server likewise automatically convert single quotes (left and right) in search strings to curly quotes.

Problem solved--Still would need to think of potential negatives.  But since no permanent data would be changed, it would be easily reversible.


Inconsistent loading order of packages and their parts

Describe the bug
It seems there is different loading order of plugins and their parts if using empty vimrc / packload / packadd  / packadd!
To Reproduce
I've created https://github.com/sheerun/vim-packages-bug that should better illustrate the problem. You can run there bash script.sh to produce following output:

Putting package in ~/.vim/pack/*/start/* without pack commands in ~/.vimrc

loading plugin/*.vim after native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

Putting package in ~/.vim/pack/*/start/* with packload

loading plugin/*.vim before native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading plugin/*.vim before native vim plugins
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

Putting package in ~/.vim/pack/*/start/* with packadd vim-packages-bug in vimrc

loading plugin/*.vim before native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading plugin/*.vim before native vim plugins
loading plugin/*.vim after native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

Putting package in ~/.vim/pack/*/start/* with packadd! vim-packages-bug in vimrc

loading plugin/*.vim before native vim plugins
loading plugin/*.vim after native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

Putting package in ~/.vim/pack/*/opt/* with packadd vim-packages-bug in vimrc

loading plugin/*.vim before native vim plugins
loading ftdetect/*.vim after native filetype.vim
loading plugin/*.vim before native vim plugins
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

Putting package in ~/.vim/pack/*/opt/* with packadd! vim-packages-bug in vimrc

loading plugin/*.vim before native vim plugins
loading after/plugin/*.vim after native vim plugins
Press ENTER or type command to continue

To summarize

plugins  can be loaded before or after vim plugins, possibly multiple times
ftdetect is loaded always after vim filetypes, possibly multiple times (all other package managers always load ftdetect before vim native files..)
In some cases no part of plugin is loaded before vim parts

Things get even worse if multiple packages are used, because then they can be loaded in alphabetical order (no pack commands), in defined order (packadd commands), or reverse defined order (packadd! commands), and all combinations of these. Order of loading is also affected if some plugins are calling  filetype plugin indent on within themselves (like vim-sensible). Also all other package managers are loading filetype.vim of plugin while vim8 packages do not, ever.
Honestly it's pretty overwhelming for someone who would like to depend on some order of loading in package manager (the vim-polyglot plugin I'm developing needs to be loaded before vim's filetype.vim, and other plugins overriding its autocommands).
Environment (please complete the following information):
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Sep 16 2020 23:47:44)
macOS version
Included patches: 1-1700
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl               -farsi             +mouse_sgr         +tag_binary
+arabic            +file_in_path      -mouse_sysmouse    -tag_old_static
+autocmd           +find_in_path      +mouse_urxvt       -tag_any_white
+autochdir         +float             +mouse_xterm       -tcl
-autoservername    +folding           +multi_byte        +termguicolors
-balloon_eval      -footer            +multi_lang        +terminal
+balloon_eval_term +fork()            -mzscheme          +terminfo
-browse            +gettext           +netbeans_intg     +termresponse
++builtin_terms    -hangul_input      +num64             +textobjects
+byte_offset       +iconv             +packages          +textprop
+channel           +insert_expand     +path_extra        +timers
+cindent           +ipv6              +perl              +title
-clientserver      +job               +persistent_undo   -toolbar
+clipboard         +jumplist          +popupwin          +user_commands
+cmdline_compl     +keymap            +postscript        +vartabs
+cmdline_hist      +lambda            +printer           +vertsplit
+cmdline_info      +langmap           +profile           +virtualedit
+comments          +libcall           -python            +visual
+conceal           +linebreak         +python3           +visualextra
+cryptv            +lispindent        +quickfix          +viminfo
+cscope            +listcmds          +reltime           +vreplace
+cursorbind        +localmap          +rightleft         +wildignore
+cursorshape       +lua               +ruby              +wildmenu
+dialog_con        +menu              +scrollbind        +windows
+diff              +mksession         +signs             +writebackup
+digraphs          +modify_fname      +smartindent       -X11
-dnd               +mouse             -sound             -xfontset
-ebcdic            -mouseshape        +spell             -xim
+emacs_tags        +mouse_dec         +startuptime       -xpm
+eval              -mouse_gpm         +statusline        -xsmp
+ex_extra          -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+extra_search      +mouse_netterm     +syntax            -xterm_save
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X -DMACOS_X_DARWIN  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang   -L. -fstack-protector-strong -L/usr/local/lib -L/usr/local/opt/libyaml/lib<EMAIL_ADDRESS>-L/usr/local/opt/readline/lib  -L/usr/local/lib -o vim        -lncurses -liconv -lintl -framework AppKit  -L/usr/local/opt/lua/lib -llua5.3 -mmacosx-version-min=10.15 -fstack-protector-strong -L/usr/local/lib  -L/usr/local/Cellar/perl/5.32.0/lib/perl5/5.32.0/darwin-thread-multi-2level/CORE -lperl -lm -lutil -lc<EMAIL_ADDRESS>-lpython3.8 -framework CoreFoundation  -lruby.2.7

Also ftdetect can be not loaded if packadd! is used and plugin is in opt (but it works if it's in start)....

Oh, plus vim and nvim have different behavior for each of these: https://github.com/k-takata/minpac/pull/118#issue-495540863 (minpac depends on vim8 / neovim packages to load plugins instead of modiifying path)
I mean in current form vim8 packages are pretty much unstable, and actually much harder to work with than any other solution available. I think they need v2.0 with much simpler logic, something like this:

No splitting of plugins of packages into start and opt, all plugins are opt, the tree of pack is simplified to: ~/.vim/pack/<packagename>/<pluginname>/{indent,plugin,ftdetect,after,...}
To load plugin, it needs to be explicitly called in ~/.vimrc (this ensures proper loading order), with some new command like pack vim-something
Directories of plugins are added to rtp at the front in chronological order, and after directories of plugins are added at the end of rtp, also in chronological order (i.e. order of packages defined by order of pack commands in ~/.vimrc).
No command like packload which can load plugins in unstable order (alphabetical, reverse alphabetical)
It's plugins that are responsible for loading their sub-dependencies, not vim (e.g. by calling pack command in their plugin/* scripts)

"the vim-polyglot plugin I'm developing needs to be loaded before vim's filetype.vim".  That is a very difficult restriction you are making.  The user may have "syntax on" somewhere you don't expect it.  Why not remove this dependency?
Also, packages should not depend on the loading order.  :packload first adds all the directories to 'runtimepath', so that when there are dependencies it should be possible to find them.

Well, for me it's difference between 60ms and 10ms of loading time, because that's how much hundreds of au! calls take ( au calls are pretty fast), and I need to run them to override vim defaults if I load after vim. I can program if to use au! instead of au if for some reason vim files loaded first (which would satisfy "packages should not depend on the loading order"), but it'll always have performance penalty. With current implementation of vim8 packages it's pretty much guaranteed that vim-polyglot loads later, while for all other plugin managers it's pretty much guaranteed that it'll always load first.

For filetypes it's basically "last one wins", so why do you need to remove what was added before?
Specifically, is this about the order in which autocommands are added, or the order of 'runtimepath', which changes in which order the plugins are found?

runtimepath works well for all files except initial ftdetect files, or rather filetype.vim (e.g. ruby.vim of vim-polyglot is properly loaded always before vim's). Indeed there's "last wins", but changing filetype multiple for loaded file has its own performance penalty, so it's best handled by ensuring correct filetype is applied in the first place (vim uses setf wchich is "first one wins").

I mean I can hack anything as long as something is loaded before vim's runtime files. Current implementation doesn't load anything at all, so there's no way around it. I basically must live with vim runtime files being sourced, and I cannot do anything about it, this is the gist of this issue.

runtimepath works well for all files except initial ftdetect files, or
rather filetype.vim (e.g. ruby.vim of vim-polyglot is properly loaded
always before vim's). Indeed there's "last wins", but changing
filetype multiple for loaded file has its own performance penalty, so
it's best handled by ensuring correct filetype is applied in the first
place (vim uses setf wchich is "first one wins").

Yeah, that's right, using "setf" is different.  And you don't really
want to set one filetype and then overrule it, causing multiple filetype
plugins to be loaded in sequence.
I would think it works to overrule the autocommand from filetype.vim:
au BufRead *.js
--- Autocommands ---
filetypedetect  BufRead
    *.js      setf javascript
au! filetypedetect BufNewFile,BufRead *.js setf myJs
au BufRead *.js
--- Autocommands ---
filetypedetect  BufRead
    *.js      setf myJs

--
How To Keep A Healthy Level Of Insanity:
6. In the memo field of all your checks, write "for sexual favors".
/// Bram Moolenaar --<EMAIL_ADDRESS>-- http://www.Moolenaar.net   \
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\\  an exciting new programming language -- http://www.Zimbu.org        ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

Yes indeed, I've mentioned I can use au! but it has heavy performance penalty. 60ms vs 10ms loading time

I've decided to not use au! for each overridden filetype, but au! filetypedetect + basically copy all vim's filetype.vim into vim-polyglot, which finally I know will load last :P
As of now the issue is twofold:

vim's filetype.vim is unnecessairly loaded which takes 25ms (it is fully overridden by vim-polyglot)
au! can't be used by other plugins because it depends on load order of plugins, and also it requires that arguments are exactly the same which they often aren't. for example one plugin can use au! BufRead Makefile.am setf make, when second uses au! BufRead [Mm]akeifile.am setf makefile.

This could all by instructing vim to source second plugin first. which currently isn't encouraged ("packages should not depend on the loading order") or even possible to enforce with vim8 packages

Yes indeed, I've mentioned I can use au! but it has heavy
performance penalty. 60ms vs 10ms loading time

I suppose that's because it has to go over all the autocommand to find
the ones that need to be deleted.  Perhaps we can make this faster, that
is useful in many ways.
So this delay happens if, after "filetype on", you use an "au!" command?
Can you give an example?
--
How To Keep A Healthy Level Of Insanity:
15. Five days in advance, tell your friends you can't attend their
party because you're not in the mood.
/// Bram Moolenaar --<EMAIL_ADDRESS>-- http://www.Moolenaar.net   \
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\\  an exciting new programming language -- http://www.Zimbu.org        ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

Adam Stankiewicz wrote:

I've decided to not use au! for each overridden filetype, but au! filetypedetect + basically copy all vim's filetype.vim into
vim-polyglot, which finally I know will load last :P
As of now the issue is twofold:

vim's filetype.vim is unnecessairly loaded which takes 25ms (it is
fully overridden by vim-polyglot)

I don't see how you can override all of filetype.vim. Do you mean to
clear it and load it again?

au! can't be used by other plugins because it depends on load
order of plugins, and also it requires that arguments are exactly the
same which they often aren't. for example one plugin can use au! BufRead Makefile.am setf make, when second uses au! BufRead [Mm]akeifile.am setf makefile.

This could all by instructing vim to source second plugin first. which
currently isn't encouraged ("packages should not depend on the loading
order") or even possible to enforce with vim8 packages

It's very tricky to make plugins load in the right order.  Especially if
you don't know what the right order is.
Autocommands are executed in the order they are defined.  If a later
plugin adds an autocommand for filetype detection, it can overrule what
happened earlier:
au BufNewFile,BufRead *.mine set filetype=mine
In filetype.vim ":setfiletype" is used, which won't change 'filetype' if
it was already set, thus it doesn't matter if it's loaded before or
after this.  The only disadvantage is that 'filetype' might be set
twice.  Is that really a problem?
If two plugins fight over the same rule, well, then there is a problem.
--
How To Keep A Healthy Level Of Insanity:
16. Have your coworkers address you by your wrestling name, Rock Hard Kim.
/// Bram Moolenaar --<EMAIL_ADDRESS>-- http://www.Moolenaar.net   \
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\\  an exciting new programming language -- http://www.Zimbu.org        ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

I don't see how you can override all of filetype.vim. Do you mean to
clear it and load it again?

In case vim's filetype.vim is sourced first, I'm running au! filetypedetect which quickly clears any autocommands set by filetype.vim, and then I "source" filetype.vim again.
In case vim's filetype.vim is sourced last, thanks to let did_load_filetypes = 1 set by vim-polyglot, it makes sourcing vim's filetype.vim a no-op 0ms, instead of 25ms (optimization which I cannot do if filetype.vim of vim is loaded first).

So this delay happens if, after "filetype on", you use an "au!" command?
Can you give an example?

It isn't one-time delay, it's difference of calling hundreds of au, vs calling analogical hundreds of au! commands (I'm no longer doing that, what I'm doing now is just single au! filetypedetect, which is fast and doesn't need to be improved).
Currently without vim-polyglot loading time is as follows:

filetype.vim = 25ms

With vim-polyglot loaded after filetype.vim loading time is as follows:

filetype.vim = 25ms
polyglot = 10ms

If vim-polyglot is loaded before filetype.vim, then loading time is as follows:

polyglot = 10ms
filetype.vim = 0ms

So using vim-polyglot makes startup time faster than native, if sourced in correct order


[BUG] - Password change is not working

Description
On the password change page, you will received this error when trying to change the password:
from ktem.pages.admin.user import validate_password
ModuleNotFoundError: No module named 'ktem.pages.admin'
Logs
File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/gradio/queueing.py", line 536, in process_events
    response = await route_utils.call_process_api(
  File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/gradio/route_utils.py", line 276, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/gradio/blocks.py", line 1923, in process_api
    result = await self.call_function(
  File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/gradio/blocks.py", line 1508, in call_function
    prediction = await anyio.to_thread.run_sync(  # type: ignore
  File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/gradio/utils.py", line 818, in wrapper
    response = f(*args, **kwargs)
  File "/home/ubuntu/.local/share/virtualenvs/kotaemon-QmUhOJgz/lib/python3.10/site-packages/ktem/pages/settings.py", line 232, in change_password
    from ktem.pages.admin.user import validate_password
ModuleNotFoundError: No module named 'ktem.pages.admin'

Browsers
No response
OS
No response
Additional information
No response

Hi, is this happened when changing the password of admin user or for a new normal user?

admin password

I will try the docker installation and see if it happens also.

Confirm this issue happened with Docker image as well. Will release a fix soon.

Same error on the docker image.


make run is unfriendly to parallel builds

At this line of code, if we execute make run -j4 then it'll incorrectly build iso and run qemu at the same time.

Do you have any solution? 😂

I'm working on using bootimage to replace the current ugly build process (maybe for x86_64 only). Wish all these problems will disappear after that.

Great

@wangrunji0408 I found another problem.
The parallel options passed to make does not go into cargo, so make -j8 iso is of no use, just the same as make iso.
Hope this will be fixed in the future.

It seems cargo already support parallel build.
So there is no need to parallel make.


commandline use more one command show error

if I input "commandline": "echo '12345';ssh<EMAIL_ADDRESS>it show 12345 and then require my password, like this

but I use the following settings
{
"closeOnExit": true,
"commandline": "echo '12345';ssh<EMAIL_ADDRESS>"guid": "{592D0432-1020-4DF6-83D4-928A7BE1D31B}",
"name": "Test"
}
it shows
[启动“echo '12345'”时出现错误 0x80070002]
[start“echo '12345'” error 0x80070002]
is this a bug ?

No I don't think this is a bug. The commandline is not the same as whatever you type inside a shell (in this case powershell). The commandline needs to be a program that can be found and spawned by the terminal. In this case you need to use something like this:
{
    "commandline": "powershell -Command echo '12345';ssh<EMAIL_ADDRESS>}

See here for the difference between a shell and a terminal. Or search online for more detailed explanation. If you're really interested , the keywords would be "shell", "terminal", "CLI" and "tty".
And BTW you should not leak sensitive information like IP addresses, just for your own sake.


Change Encoding format on XML Header

Hello,
I have used option xmlHeader: true, which gives <?xml version="1.0" encoding="utf-8"?> on top of the XML.
However, I want to change "utf-8" to something else like "TIS-620", how do I do that? Thank you.

hey. you should add an argument here https://github.com/soldair/node-jsontoxml/blob/master/jsontoxml.js#L118
and submit a pull request =)


upd(brave-browser-deb):1.61.114->1.63.162

latest release

For future reference, use one set of backticks instead of two for the commit.


Access Groups support for OIDC claims

Current Terraform and Cloudflare provider version
Terraform v1.5.1
on darwin_arm64

provider registry.terraform.io/cloudflare/cloudflare v3.35.0

Description
Support for OIDC claims in resource cloudflare_access_group. Currently, it supports just about every other method with the exception of OIDC claims.
Use cases
It's useful to be able to use OIDC claims in access groups where required, and be able to do it using Terraform.
Potential Terraform configuration
resource "cloudflare_access_group" "group" {
  account_id = var.cloudflare_account_id
  name       = "application access"

  include {
    oidc_claim {
      name = "oidc claim name"
      value = "oidc claim name"
    }
  }
}

References
No response

This issue has been closed as we are now tracking this internally with service teams directly. If you would like an update or to be notified when/if the product ships with this change, please reach out to Cloudflare Support or your account team who can watch the internal feature request for you.


Position inconsistency

Please provide the environment you discovered this bug in.
I noticed this bug when working with spartan.ng sonner, which is just a wrapper of this library. Here's a minimal repo.
Description
There seems to be some inconsistencies of the direction which the sonner is coming from. By default the position is "bottom-right", making a toast callback with any top position makes it so that the direction is opposite (instead of appearing top to bottom, it's appearing bottom to top). The effect is also the same if the position of ngx-sonner is set to top, and the toast position is set to bottom.
Please provide the exception or error you saw
No response
Other information
No response
I would be willing to submit a PR to fix this issue

[ ] Yes
[X] No

Thank you for reporting this! I will take a look into it a soon as I can

:tada: This issue has been resolved in version 2.0.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add allowedEncodings option (#30857)

Fixes #30857

Thanks, I went with a slightly different change: https://github.com/Microsoft/vscode/commit/9c882c7c93b605e23870e77adf52236a138ee534

I can't get the files.restrictGuessedEncodings option to work in VSCode version 1.33.1. When will this be released?


Force parentheses between unary op and binary power.

Fixes #646. This patch forces expressions containing unary and power ops, like this...
+a ** -b ** ~c ** -d  # This is legal blackened code.

...to be reformatted with parentheses to make operator precedence clear, like this...
+(a ** -(b ** ~(c ** -d)))  # This is legal blackened code.

The two remaining mypy errors are due to #907, unrelated.

Is there anything more I need to do for this? This is my first time contributing, so I'm not sure if I missed something!

Thanks!


Connection Issue

Has anybody experienced connection issues recently?
Here is what I get after successfully loggin in:
  spotify-web POST "https://play.spotify.com/xhr/json/auth.php" +0ms
  spotify-web auth 200 status code, "application/json; charset=utf-8" content-type +861ms
  spotify-web ap resolver {"hostname":"apresolve.spotify.com","site":null,"override":null} +1ms
  spotify-web GET "http://apresolve.spotify.com" +0ms
  spotify-web ap resolver 200 status code, "application/json; charset=utf-8" content-type +742ms
  spotify-web WS "wss://lon2-linkap-a5.ap.spotify.com:443/" +4ms
  spotify-web WebSocket "open" event +1s
  spotify-web connect() +0ms
  spotify-web sendCommand("connect", [...]) +0ms
  spotify-web storing callback function for message id 0 +1ms
  spotify-web sending command: {"name":"connect","id":"0","args":[..]} +0ms
  spotify-web WebSocket "message" event: {"id":0,"result":"ok"} +227ms
  spotify-web starting heartbeat every 180 seconds +1ms
  spotify-web WebSocket "message" event: {"message":["do_work","var t2; try{ var d=new Date(); t2=d[(35725129862..toString(36<<0))[0]];}catch(_) {};this.reply('' + t2);"]} +2ms
spotify-web got "do_work" payload: "var t2; try{ var d=new Date(); t2=d[(35725129862..toString(36<<0))[0]];}catch(_) {};this.reply('' + t2);" +0ms
  spotify-web reply(["undefined"]) +1ms
  spotify-web sendCommand("sp/work_done", ["undefined"]) +0ms
  spotify-web storing callback function for message id 1 +0ms
  spotify-web sending command: {"name":"sp/work_done","id":"1","args":["undefined"]} +0ms
  spotify-web WebSocket "message" event: {"message":["ping_flash2","145 46 160 145 146 206 188 171 82 230 119 80 20 244 126 60 21 215 69 147"]} +3ms

{ [Error: connect ECONNREFUSED]
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect' }

Error: connect ECONNREFUSED
  at errnoException (net.js:901:11)
  at Object.afterConnect [as oncomplete] (net.js:892:19)

I can't login since today :/

@1994rstefan I think your http://ping-pong.spotify.nodestuff.net service is down :frowning:

:+1: me either

@TooTallNate Are you sure it's because of the ping-pong server? I ran his server on my computer and connected it with the spotify.js but I still can't login

@Zazama beat me to the punch, was going to say the same thing!

Well that's definitely what the error log above is from. I haven't tried running the server manually.

Tried running the server manually, it doesn't crash.  But i'm never getting the "login_complete" event in spotify-web.

Sorry for the long delay - my server is running again now.
I dont know why it was down, looks like there just crashed something...
Hope it works again now
Ragards

Hurray! Everything's working again.

Nice! Awesome work guys!

Why is a connection to your server required?

Thank you guys!!!

@art0rz
node-spotify-web uses the same API to Spotify as the official Flash webplayer does. The process of logging in to spotify includes some security functions. Spotify sends a "ping" to the client (this ping is just a string that could for example look like this: 236 162 128 247 137 222 96 1 77 9 90 35 23 180 159 101 19 126 232 254). The Client (more detailed the Flash running at the client) then generates a "pong" based on the ping (the pong is also just a string, for the above mentioned ping it would be: 155 89 69 186 47 172 243 69 97 9) and sends it back to the Spotify API.
How this pong is generated is a secret and also changed in the past.
Because there is Flash required for generating this pongs I provide as solution an API for this task. My server runs a webbrowser and the Flash, using the API you can indirectly access the official function of the Spotify webplayer Flash for generating the pong.
So the reason why a connection is required is simply for generating this pong string.
You can also run your own API if you want. See https://github.com/1994rstefan/Spotify-Ping-Pong-API for more information about that.

Thank you for the clarification.
Can I assume no login credentials or session cookies sent through your service?

No credentials are transfered, only the ping gets send.
See the sendPong function for more details: https://github.com/TooTallNate/node-spotify-web/blob/master/lib/spotify.js#L476

@1994rstefan is your site down again? It's working if I set your server up locally, but not with your "http://ping-pong.spotify.nodestuff.net/" server

Sorry, up again.

Would it be possible to host this server locally using slimerjs? That way it's not dependent on a central service.

Not sure why slimerjs would be needed. The node-spotify-web could make a dependency for Spotify-Ping-Pong-API, then when node-spotify-web is required on your app it init's the local server and then this local server is used instead.

Actually after looking into it more you have to have the worker.html page running in a browser on the server. The browser that it runs in has to support Adobe Flash, so this is probably why slimerjs or PhantomJS would not work. I'll look into it tomorrow and if I can get it working I will submit a pull request.

I guess that it would be a good idea to host the api on more than one server. The client could then randomly choose one to connect to.
The best would of course be if we could reverse engineer the flash file again and just code it in javascript. But until that happens I don't think that everyone wants to run their own server since that requires running an instance of Flash and a full web browser.

So I got it working with slimerjs but the problem is that it has to open a window and leave it running for it to work. So this will not work in an environment without a graphical interface. There is a tool called xvfb that will run slimerjs headless but this only works on Linux and OSX. With this API we want it to be cross-platform and not require an additional window to be open in order for it to run. I think at this point the current solution is probably the best.
@LinusU, I agree that having multiple servers that it trys to ping-pong with would be optimal for redundancy. If anyone has any extra servers that can use towards this that would be awesome :).

I could set my vps on to provide the service :)

Can we use xvfb where it is supported, and fallback to the service if it's
not?
On Thu, Jan 29, 2015 at 9:18 AM, Denys Vitali<EMAIL_ADDRESS>wrote:

I could set my vps on to provide the service :)
—
Reply to this email directly or view it on GitHub
https://github.com/TooTallNate/node-spotify-web/issues/102#issuecomment-72053204
.

Tried it, but without success. Any suggestion? Firefox starts just fine, but hasn't Flash plugin in it, neither by installing Pepper Flash it works...

@denysvitali
My server is running a minimal Debian Wheezy (64 Bit) and i had to install following additional packages: nodejs, xvfb, firefox and flashplugin-nonfree
(Maybe you have to enable the nonfree source list)
After creating a directory inside my home dir (~/spotify-ping-pong-api/) and cd into it and cloning the repository (so the sever.js is in the current directory) you have to run the following commands:
screen nodejs server.js
// exit screen with pressing strg+a and then d
screen Xvfb :0 -screen 0 1024x768x24
// exit screen with pressing strg+a and then d
DISPLAY=:0 screen firefox http://localhost:8080/worker.html
That should be enough to get it up running.
If you want to access the graphicall interface you can do so with VNC.
Install the package x11vnc, run "x11vnc -display :0" and connect to your IP via VNC.
If you close the VNC connection x11vnc will also stop, so nobody else can connect.
I am not sure if I changed any settings in Firefox via VNC, but I do not think i did.

That's why! I installed pepper flash instead of flashplugin-nonfree
Here you are:
http://vps.denv.it:1335/12-34-56-78-90-12-34-56-78-90-12-34-56-78-90-12-34-56-78-90

Back to not working again.  stopped last night.

Hello,
I am sorry that the service was down again.
To solve this problem for the future I have now set up two independent API-Servers.
The domain ping-pong.spotify.nodestuff.net now has two A records pointing to both of the servers, but this is not perfect because its random to which server the request is sent.
I also configured the domains 1.ping-pong.spotify.nodestuff.net and 2.ping-pong.spotify.nodestuff.net to point respectively to server one and two. It would be nice if someone could create a pull request to use the two servers in a usefull way. (Try to request server 1, if it failes, try to request server 2)
Best regards

Looks like the service is down again. Tried to set up my local ping pong server and everything works, but I always get errors because I live in germany. Screw that.

@ThomasHerzog I'm also from Germany using my own ping pong server without problems. I'm using node.js 0.10.36. I've had region errors when using version 0.11.* or 0.12.*, so I just fell back to the old one.

@Zazama Thank you very much! I got it running using the older node.js version!


Exam participation mode

Checklist

[x] I tested all changes and all related features with different users (student, tutor, instructor, admin) locally.
[x] Client: I documented the TypeScript code using JSDoc style.
[x] Client: I added multiple screenshots/screencasts of my UI changes
[x] Client: I translated all the newly inserted strings into German and English

Motivation and Context

First iteration of offline mode -> prevent that users cannot continue working on their exam if they loaded all application data (module.js) and have downloaded the StudentExam at one point
Description

Steps for Testing

Setup

Log in to Artemis
Create an exam with text exercises (currently only text-exercises and quiz-exercises supported, careful: you have to set a dueDate for each exercise as well), set a reasonable start and end date (you'll have to wait until it's over)
Add students/yourself to the Exam
Generate the StudentExams
Generate the participations and submissions
Navigate to the Exam (CourseOverview -> Course -> Exams)
Participate in the Exam by clicking on it

Navigate through exam

Accept the confirmation text and click on start exam-> A view opens with a navigation header + the first exercise of the exam
You can navigate throught the exam by using the navigation bar at the top
You can navigate by clicking on save and continue button (submitts current state and goes to next exercise)

Saving of submissions
Using header bar

Change the exercise
navigate to the next exercise, using the header bar -> submission should be saved to server

Using save and continue

Change the exercise
Click save and continue -> submission should be saved to the server

Automatic update every 60seconds

Change the exercise
Wait for 60seconds -> submission should be saved to the server
If a update is missed, because you were offline, the server should update it in the next cycle / the next time a save event is triggered (using header bar, using save and continue or after 60seconds)

SImple offline mode

Once loaded, the exam can be worked on even when you get offline.
Once you get online again, your last submission of each exercise you worked on is synched to the server (by triggering a save event)

Other

Check that the request time is displayed correctly
After the exam finishes, you should be navigated to the summary screen

Screenshots

Accept the confirmation text and click on start exam-> A view opens with a navigation header + the first exercise of the exam

This works as expected, but when I tried this, I was immediately greeted with a prompt saying that my name was incorrectly entered. The prompt seems to stay until I click the page (and ignore changes to the text field), but I'm not entirely sure.

You can navigate throught the exam by using the navigation bar at the top
You can navigate by clicking on save and continue button (submitts current state and goes to next exercise)
navigate to the next exercise, using the header bar -> submission should be saved to server
Click save and continue -> submission should be saved to the server

These all work as intended.
I have not tested the offline mode or the automatic 60s submission yet.

After the exam finishes, you should be navigated to the summary screen

This works.
Other things I've noticed:
I cannot delete exams anymore after participating in them (note: I did not check whether this is caused by actually participating or whether this is a result of generating the student exams / participations), instead, I get an error:
translation-not-found[Http failure response for https://artemistest.ase.in.tum.de/api/courses/263/exams/94: 500 OK]

The exam timer appears to be 60 seconds above what the exam's configuration specifies, e.g. if I create an exam that lasts 1 minute, the timer starts at "1:60" (I'm not sure why this is not 2:00, but this is the way I've observed it).

Timer issue should be fixed

This is not restricted to this pull request.
It seems that currently, it is not possible to add exercises to exercise groups.
Bug that currently prevents testing
Steps to reproduce: 1. Create an exam 2. Create an exercise group for that exam 3. Create a new text exercise (I did not check if this is only restricted to text exercises) for that group 4. It will say that a new exercise has been created, but it will not show up in the corresponding exercise group (Expected: The new exercise shows up in the exercise group).

Sascha is currently fixing the issue in a separte branch which will be merged against develop. After that everything should work again

Tested it afterwards on the develop:

When reloading during the exam I always get on the confirmation page and have to enter the Name again. Sure usually you don't have to reload but we should save the confirmation of the student and don't ask him again. The confirmation is useless if we don't save that he clicked it
During offline mode not saved exercises will show up yellow. This is not the case in the online mode. Therefore if you don't disconnect you never know if your progress is saved or not yet. => The exercise index should be yellow in online mode if not yet saved to the server
Why don't you have numbers as exercise indexes but the Edit-Icons? I'd prefer the numbers or alternating icons depending on the state and progress because if you have 10 icons next to each other which are the same and have no meaning you can even leave them out