Red-zone Integrity

It was a pleasure chatting with the authors at OSDI/ATC! We are currently considering integrating zpoline into Gramine (formerly known as Graphene): https://github.com/gramineproject/gramine
Just wondering if you have any thought on replacing the syscall instruction with call when the red zone (128-byte region below the RSP) is used, likely in the leaf functions. Does this ever occur in your experiment? If so, how did you handle this case?
Thank you!

Thank you for your message.
It was great for me to have a chat with you at the conference.
Regarding the issue, I am sorry, but we did not know about the red zone while we were writing the paper.
As reported by a previous issue, our implementation causes faults in some cases; now, I think it would be because of the red zone.
I will try to resolve this issue and update the implementation once I find a solution.
Thank you for letting me know about this issue.

Thank you so much for the response. I understand that conducting research can be difficult and sometimes there are unforeseen issues.
On top of my head, one (not so ideal) solution is to replace "syscall" with "jmp *eax" instead of "call *eax", but only serve one code spot at a time. For the other "syscall" locations, they can be replaced some illegal instruction, causing an exception to be capture (this naturally happens on SGX) and swap the serving target.
To assess this idea, just wondering if you have any statistics about normally how many "syscall" instructions exist in a static binary and how often they are called during execution.
Thank you!

Thank you very much for your great idea.
Regarding the number of syscall instructions in binaries, I found there are 544 in glibc-2.35 and 50 in ld.so.
I checked the frequency of system call invocation by counting the number of system calls triggered in one second during the simple HTTP server and Redis experiments; The following shows the results.
Simple HTTP server

system call
number of invocation in one second

read
1185497

write
1185497

epoll_wait
1185497

Redis

system call
number of invocation in one second

read
665983

write
665973

close
10

epoll_wait
665997

openat
10

The server programs execute the loop of epoll_wait, read, write system calls, and we may be able to guess what would come next at runtime, while it would depend on applications.
I deeply appreciate that you bring the idea to address the issue.

Thanks for the data! Given the amount of syscall instructions and the frequency of system calls, swapping the jmp target may be kinds of expensive.
One optimization I can think of is that we can detect whether the syscall instruction is in a leaf function (which will not use the red zone). We only have to use "jmp *eax" in a leaf function and that should reduce the amount of the syscall instructions that need to be handled with exception handler.

Thank you very much for your idea; it quite makes sense.
Please let me try the approach you introduced, and I will get back here when I have an update.
Again, I really appreciate your kind help in resolving the issue.

As a temporary workaround, compiling with -mno-red-zone (gcc) could be useful ?
Note that every component/library used by program must be compiled with -mno-red-zone. This could be made fully self contained if you are able to build a statically compiled version of the program with every component compiled with -mno-red-zone.
It would be useful to see if a crashing program starts working when compiled with -mno-red-zone.
But then again, if your program consists of compilers other than C/C++ that use the red-zone or hand-written assembly that uses that red-zone even this workaround may not be fully helpful.

I believe your workaround, compiling with -mno-red-zone, is definitely useful though the limitation you describe remains.
Thank you very much for providing the information.

Using -mno-red-zone could be a good way to verify if the use of red zone is indeed the root cause of recently reported crashes.
In general, I think the assumption is that the users cannot or do not want to recompile the binaries, and thus we have to resort to dynamic binary modification for intercepting system calls. If recompilation is possible, we won't have this problem, since we can just replace system calls at the source code or IR level, or add more empty bytes to the instruction to allow modification.

Maybe.. add a pop %rcx before call *%rax and add 0x60; int3 after it... This allow the fix of red zone using value in rcx.
The seq is pop %rcx; call *%rax; 0x60; int3; = 0x59 0xff 0xd0 0x60 0xcc.
This allows illegal instructions if jump to the byte +2, +3, +4. but the fix-up do have some complexity...
Will require %rax to be small enough so 0xd0 0x60 0xcc = shlb 0xcc(%rax) will crash as expected if jump to byte +2..
And if it is not crash... at least the shifted bit will be keeped inside CF, so the next int3 can raise signal and the modified bit can be restored using value in CF.
Because ModR/M byte must be 0x10 ([rax]), 0x50 ([rax]+disp8), 0x90 ([rax]+disp32) or 0xD0 (rax) to be related to rax... 0x10 and 0x50 will be adc and push, not as good as shlb.. 0x90 will cause a [rax]+disp32 which is even more uncontrollable (given the endianness)..
And it is anyway still better than normal 5 bytes direct jmp because that will contain an uncontrollable constant distance and may cause problem if any those bytes are jump targets which may require rewrite instruction before or even impossible to rewrite.

@Yangff
Thank you very much for your very cool idea.
I have experimentally implemented some foundational parts, and please let me share the current status.
patch
The following is the diff to commit 0a349e65c102f8f9bdbbf6da0a52c4006589178b .
diff --git a/main.c b/main.c
index 281a235..a9376ff 100644
--- a/main.c
+++ b/main.c
@@ -24,6 +24,7 @@
 #include <stdbool.h>
 #include <stdlib.h>
 #include <string.h>
+#include <signal.h>
 #include <errno.h>
 #include <assert.h>
 #include <sys/syscall.h>
@@ -111,6 +112,16 @@ static bool is_replaced_instruction_addr(uintptr_t addr)

 #endif

+#define MAX_REPLACED_SYSCALL (2048)
+
+struct replaced_syscall {
+	uintptr_t addr;
+	uint8_t code[6 + 16 + 12];
+};
+
+static struct replaced_syscall *replaced_syscall_list = NULL;
+static unsigned int replaced_sycall_cnt = 0;
+
 extern void syscall_addr(void);
 extern long enter_syscall(int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t);
 extern void asm_syscall_hook(void);
@@ -156,18 +167,6 @@ void ____asm_impl(void)
 	"asm_syscall_hook: \n\t"
 	"popq %rax \n\t" /* restore %rax saved in the trampoline code */

-	/* discard pushed 0x90 for 0xeb 0x6a 0x90 if rax is n * 3 + 1 */
-	"pushq %rdi \n\t"
-	"pushq %rax \n\t"
-	"movabs $0xaaaaaaaaaaaaaaab, %rdi \n\t"
-	"imul %rdi, %rax \n\t"
-	"cmp %rdi, %rax \n\t"
-	"popq %rax \n\t"
-	"popq %rdi \n\t"
-	"jb skip_pop \n\t"
-	"addq $8, %rsp \n\t"
-	"skip_pop: \n\t"
-
 	"cmpq $15, %rax \n\t" // rt_sigreturn
 	"je do_rt_sigreturn \n\t"
 	"pushq %rbp \n\t"
@@ -188,10 +187,10 @@ void ____asm_impl(void)
 	"pushq %rdi \n\t"
 	"pushq %rsi \n\t"
 	"pushq %rdx \n\t"
-	"pushq %rcx \n\t"

 	/* arguments for syscall_hook */

+	"pushq %rbp \n\t"
 	"pushq 8(%rbp) \n\t"	// return address
 	"pushq %rax \n\t"
 	"pushq %r10 \n\t"
@@ -201,9 +200,8 @@ void ____asm_impl(void)
 	"callq syscall_hook \n\t"

 	"popq %r10 \n\t"
-	"addq $16, %rsp \n\t"	// discard arg7 and arg8
+	"addq $24, %rsp \n\t"	// discard arg7 and arg8 and rcx

-	"popq %rcx \n\t"
 	"popq %rdx \n\t"
 	"popq %rsi \n\t"
 	"popq %rdi \n\t"
@@ -213,10 +211,14 @@ void ____asm_impl(void)

 	"leaveq \n\t"

-	"retq \n\t"
+	"popq %rcx \n\t"
+	"addq $128, %rsp \n\t"
+
+	"jmpq *%rcx \n\t"

 	"do_rt_sigreturn:"
-	"addq $8, %rsp \n\t"
+	"popq %rcx \n\t"
+	"addq $128, %rsp \n\t"
 	"jmp syscall_addr \n\t"
 	);
 }
@@ -225,19 +227,23 @@ static long (*hook_fn)(int64_t a1, int64_t a2, int64_t a3,
 		       int64_t a4, int64_t a5, int64_t a6,
 		       int64_t a7) = enter_syscall;

+static unsigned int init_hook_cnt = 0;
+
 long syscall_hook(int64_t rdi, int64_t rsi,
 		  int64_t rdx, int64_t __rcx __attribute__((unused)),
 		  int64_t r8, int64_t r9,
 		  int64_t r10_on_stack /* 4th arg for syscall */,
 		  int64_t rax_on_stack,
-		  int64_t retptr)
+		  int64_t retptr,
+		  int64_t rbp)
 {
+	init_hook_cnt++;
 #ifdef SUPPLEMENTAL__REWRITTEN_ADDR_CHECK
 	/*
 	 * retptr is the caller's address, namely.
 	 * "supposedly", it should be callq *%rax that we replaced.
 	 */
-	if (!is_replaced_instruction_addr(retptr - 2 /* 2 is the size of syscall/sysenter */)) {
+	if (!is_replaced_instruction_addr(retptr - 2 - 1 /* 2 is the size of syscall/sysenter */)) {
 		/*
 		 * here, we detected that the program comes here
 		 * without going through our replaced callq *%rax.
@@ -261,12 +267,25 @@ long syscall_hook(int64_t rdi, int64_t rsi,
 		}
 	}

+	{
+		unsigned int i;
+		for (i = 0; i < replaced_sycall_cnt; i++) {
+			if (retptr == (int64_t) replaced_syscall_list[i].addr) {
+				*((int64_t *)(rbp + 8)) = (uintptr_t) replaced_syscall_list[i].code;
+				break;
+			}
+		}
+		assert(i != replaced_sycall_cnt);
+	}
+
 	return hook_fn(rax_on_stack, rdi, rsi, rdx, r10_on_stack, r8, r9);
 }

 struct disassembly_state {
 	char *code;
 	size_t off;
+	uintptr_t ptr;
+	uint16_t inst_off;
 };

 /*
@@ -295,13 +314,48 @@ static int do_rewrite(void *data, const char *fmt, ...)
 			 */
 			goto skip;
 		}
-		ptr[0] = 0xff; // callq
-		ptr[1] = 0xd0; // *%rax
+		s->ptr = (uintptr_t) ptr;
 #ifdef SUPPLEMENTAL__REWRITTEN_ADDR_CHECK
 		record_replaced_instruction_addr((uintptr_t) ptr);
 #endif
 	}
 skip:
+	if (s->ptr == ((uintptr_t) s->code) + s->off - 2)
+		s->inst_off = 2;
+	else if (s->inst_off) {
+		struct replaced_syscall *rc = &replaced_syscall_list[replaced_sycall_cnt];
+		unsigned int inst_size = ((uintptr_t) s->code) + s->off - s->ptr - s->inst_off;
+		assert(s->inst_off - 2 + inst_size + 12 <= sizeof(rc->code));
+		memcpy(&rc->code[s->inst_off - 2], (void *) (s->ptr + s->inst_off), inst_size); /* save original code */
+		s->inst_off += inst_size;
+		if (6 <= s->inst_off) {
+			// 48 b9 [64-bit addr (8-byte)]   movabs [asm_syscall_hook],%rcx
+			rc->code[s->inst_off - 2 + 0x0] = 0x48;
+			rc->code[s->inst_off - 2 + 0x1] = 0xb9;
+			rc->code[s->inst_off - 2 + 0x2] = ((uint64_t) (s->ptr + s->inst_off) >> (8 * 0)) & 0xff;
+			rc->code[s->inst_off - 2 + 0x3] = ((uint64_t) (s->ptr + s->inst_off) >> (8 * 1)) & 0xff;
+			rc->code[s->inst_off - 2 + 0x4] = ((uint64_t) (s->ptr + s->inst_off) >> (8 * 2)) & 0xff;
+			rc->code[s->inst_off - 2 + 0x5] = ((uint64_t) (s->ptr + s->inst_off) >> (8 * 3)) & 0xff;
+			rc->code[s->inst_off - 2 + 0x6] = ((uint64_t) (s->ptr + s->inst_off) >> (8 * 4)) & 0xff;
+			rc->code[s->inst_off - 2 + 0x7] = ((uint64_t) (s->ptr + s->inst_off) >> (8 * 5)) & 0xff;
+			rc->code[s->inst_off - 2 + 0x8] = ((uint64_t) (s->ptr + s->inst_off) >> (8 * 6)) & 0xff;
+			rc->code[s->inst_off - 2 + 0x9] = ((uint64_t) (s->ptr + s->inst_off) >> (8 * 7)) & 0xff;
+			// ff e1                   jmp    *%rcx
+			rc->code[s->inst_off - 2 + 0xa] = 0xff;
+			rc->code[s->inst_off - 2 + 0xb] = 0xe1;
+			/* overwrite */
+			((uint8_t *) s->ptr)[0] = 0x59; // pop %rcx
+			((uint8_t *) s->ptr)[1] = 0xff; // callq
+			((uint8_t *) s->ptr)[2] = 0xd0; // *%rax
+			((uint8_t *) s->ptr)[3] = 0x60; //
+			((uint8_t *) s->ptr)[4] = 0xcc; // int3
+			((uint8_t *) s->ptr)[5] = 0xcc; // int3
+			rc->addr = s->ptr + 3;
+			memset((void *) rc->addr + 3, 0xcc, s->inst_off - 6);
+			s->inst_off = 0;
+			assert(++replaced_sycall_cnt < MAX_REPLACED_SYSCALL);
+		}
+	}
 	va_end(arg);
 	return 0;
 }
@@ -422,50 +476,9 @@ static void setup_trampoline(void)
 	}

 	{
-		/*
-		 * optimized instructions to slide down
-		 * repeat of 0xeb 0x6a 0x90
-		 *
-		 * case 1 : jmp to n * 3 + 0
-		 * jmp 0x6a
-		 * nop
-		 * jmp 0x6a
-		 * nop
-		 *
-		 * case 2 : jmp to n * 3 + 1
-		 * push 0x90
-		 * jmp 0x6a
-		 * nop
-		 * jmp 0x6a
-		 *
-		 * case 3 : jmp to n * 3 + 2
-		 * nop
-		 * jmp 0x6a
-		 * nop
-		 * jmp 0x6a
-		 *
-		 * for case 2, we discard 0x90 pushed to stack
-		 *
-		 */
 		int i;
-		for (i = 0; i < NR_syscalls; i++) {
-			if (NR_syscalls - 0x6a - 2 < i)
-				((uint8_t *) mem)[i] = 0x90;
-			else {
-				int x = i % 3;
-				switch (x) {
-				case 0:
-					((uint8_t *) mem)[i] = 0xeb;
-					break;
-				case 1:
-					((uint8_t *) mem)[i] = 0x6a;
-					break;
-				case 2:
-					((uint8_t *) mem)[i] = 0x90;
-					break;
-				}
-			}
-		}
+		for (i = 0; i < NR_syscalls; i++)
+			((uint8_t *) mem)[i] = 0x90;
 	}

 	/*
@@ -473,35 +486,68 @@ static void setup_trampoline(void)
 	 *
 	 * here we embed the following code.
 	 *
+	 * sub    $0x80,%rsp
+	 * push   0x80(%rsp)
+	 * mov    %rcx,0x88(%rsp)
 	 * push   %rax
 	 * movabs [asm_syscall_hook],%rax
 	 * jmpq   *%rax
 	 *
 	 */

+	// 48 81 ec 80 00 00 00    sub    $0x80,%rsp
+	((uint8_t *) mem)[NR_syscalls + 0x00] = 0x48;
+	((uint8_t *) mem)[NR_syscalls + 0x01] = 0x81;
+	((uint8_t *) mem)[NR_syscalls + 0x02] = 0xec;
+	((uint8_t *) mem)[NR_syscalls + 0x03] = 0x80;
+	((uint8_t *) mem)[NR_syscalls + 0x04] = 0x00;
+	((uint8_t *) mem)[NR_syscalls + 0x05] = 0x00;
+	((uint8_t *) mem)[NR_syscalls + 0x06] = 0x00;
+
+	/* save retptr of callq *%rax */
+	// ff b4 24 80 00 00 00    push   0x80(%rsp)
+	((uint8_t *) mem)[NR_syscalls + 0x07] = 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x08] = 0xb4;
+	((uint8_t *) mem)[NR_syscalls + 0x09] = 0x24;
+	((uint8_t *) mem)[NR_syscalls + 0x0a] = 0x80;
+	((uint8_t *) mem)[NR_syscalls + 0x0b] = 0x00;
+	((uint8_t *) mem)[NR_syscalls + 0x0c] = 0x00;
+	((uint8_t *) mem)[NR_syscalls + 0x0d] = 0x00;
+
+	/* repair 1st 8-byte of the stack */
+	// 48 89 8c 24 88 00 00 00 mov    %rcx,0x88(%rsp)
+	((uint8_t *) mem)[NR_syscalls + 0x0e] = 0x48;
+	((uint8_t *) mem)[NR_syscalls + 0x0f] = 0x89;
+	((uint8_t *) mem)[NR_syscalls + 0x10] = 0x8c;
+	((uint8_t *) mem)[NR_syscalls + 0x11] = 0x24;
+	((uint8_t *) mem)[NR_syscalls + 0x12] = 0x88;
+	((uint8_t *) mem)[NR_syscalls + 0x13] = 0x00;
+	((uint8_t *) mem)[NR_syscalls + 0x14] = 0x00;
+	((uint8_t *) mem)[NR_syscalls + 0x15] = 0x00;
+
 	/*
 	 * save %rax on stack before overwriting
 	 * with "movabs [asm_syscall_hook],%rax",
 	 * and the saved value is resumed in asm_syscall_hook.
 	 */
 	// 50                      push   %rax
-	((uint8_t *) mem)[NR_syscalls + 0x0] = 0x50;
+	((uint8_t *) mem)[NR_syscalls + 0x16] = 0x50;

 	// 48 b8 [64-bit addr (8-byte)]   movabs [asm_syscall_hook],%rax
-	((uint8_t *) mem)[NR_syscalls + 0x1] = 0x48;
-	((uint8_t *) mem)[NR_syscalls + 0x2] = 0xb8;
-	((uint8_t *) mem)[NR_syscalls + 0x3] = ((uint64_t) asm_syscall_hook >> (8 * 0)) & 0xff;
-	((uint8_t *) mem)[NR_syscalls + 0x4] = ((uint64_t) asm_syscall_hook >> (8 * 1)) & 0xff;
-	((uint8_t *) mem)[NR_syscalls + 0x5] = ((uint64_t) asm_syscall_hook >> (8 * 2)) & 0xff;
-	((uint8_t *) mem)[NR_syscalls + 0x6] = ((uint64_t) asm_syscall_hook >> (8 * 3)) & 0xff;
-	((uint8_t *) mem)[NR_syscalls + 0x7] = ((uint64_t) asm_syscall_hook >> (8 * 4)) & 0xff;
-	((uint8_t *) mem)[NR_syscalls + 0x8] = ((uint64_t) asm_syscall_hook >> (8 * 5)) & 0xff;
-	((uint8_t *) mem)[NR_syscalls + 0x9] = ((uint64_t) asm_syscall_hook >> (8 * 6)) & 0xff;
-	((uint8_t *) mem)[NR_syscalls + 0xa] = ((uint64_t) asm_syscall_hook >> (8 * 7)) & 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x17] = 0x48;
+	((uint8_t *) mem)[NR_syscalls + 0x18] = 0xb8;
+	((uint8_t *) mem)[NR_syscalls + 0x19] = ((uint64_t) asm_syscall_hook >> (8 * 0)) & 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x1a] = ((uint64_t) asm_syscall_hook >> (8 * 1)) & 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x1b] = ((uint64_t) asm_syscall_hook >> (8 * 2)) & 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x1c] = ((uint64_t) asm_syscall_hook >> (8 * 3)) & 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x1d] = ((uint64_t) asm_syscall_hook >> (8 * 4)) & 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x1e] = ((uint64_t) asm_syscall_hook >> (8 * 5)) & 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x1f] = ((uint64_t) asm_syscall_hook >> (8 * 6)) & 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x20] = ((uint64_t) asm_syscall_hook >> (8 * 7)) & 0xff;

 	// ff e0                   jmpq   *%rax
-	((uint8_t *) mem)[NR_syscalls + 0xb] = 0xff;
-	((uint8_t *) mem)[NR_syscalls + 0xc] = 0xe0;
+	((uint8_t *) mem)[NR_syscalls + 0x21] = 0xff;
+	((uint8_t *) mem)[NR_syscalls + 0x22] = 0xe0;

 	/*
 	 * mprotect(PROT_EXEC without PROT_READ), executed
@@ -542,6 +588,80 @@ static void load_hook_lib(void)
 	}
 }

+static void sig_h(int sig, siginfo_t *si __attribute__((unused)), void *ptr)
+{
+	ucontext_t *uc = (ucontext_t *) ptr;
+	switch (sig) {
+	case SIGSEGV:
+		{
+			unsigned int i;
+			for (i = 0; i < replaced_sycall_cnt; i++) {
+				if (replaced_syscall_list[i].addr - 1 <= (uint64_t) uc->uc_mcontext.gregs[REG_RIP]
+						&& (uint64_t) uc->uc_mcontext.gregs[REG_RIP] <= replaced_syscall_list[i].addr + 3) {
+					uc->uc_mcontext.gregs[REG_RIP] = (uint64_t) &replaced_syscall_list[i].code[0];
+					break;
+				}
+			}
+			assert(i != replaced_sycall_cnt);
+		}
+		break;
+	case SIGILL:
+		{
+			unsigned i;
+			for (i = 0; i < replaced_sycall_cnt; i++) {
+				if (replaced_syscall_list[i].addr <= (uint64_t) uc->uc_mcontext.gregs[REG_RIP]
+						&& (uint64_t) uc->uc_mcontext.gregs[REG_RIP] <= replaced_syscall_list[i].addr + 3) {
+					uc->uc_mcontext.gregs[REG_RIP] = (uint64_t) &replaced_syscall_list[i].code[1];
+					break;
+				}
+			}
+			assert(i != replaced_sycall_cnt);
+		}
+		break;
+	case SIGTRAP:
+		{
+			unsigned i;
+			for (i = 0; i < replaced_sycall_cnt; i++) {
+				if (replaced_syscall_list[i].addr <= (uint64_t) uc->uc_mcontext.gregs[REG_RIP]
+						&& (uint64_t) uc->uc_mcontext.gregs[REG_RIP] <= replaced_syscall_list[i].addr + 3) {
+					uc->uc_mcontext.gregs[REG_RIP] = (uint64_t) &replaced_syscall_list[i].code[uc->uc_mcontext.gregs[REG_RIP] - replaced_syscall_list[i].addr];
+					break;
+				}
+			}
+			assert(i != replaced_sycall_cnt);
+		}
+		break;
+	}
+}
+
+static void setup_signal(void)
+{
+	{ /* SIGSEGV */
+		struct sigaction sa = { 0 };
+		sa.sa_flags = SA_SIGINFO | SA_RESTART;
+		sa.sa_sigaction = sig_h;
+		assert(!sigemptyset(&sa.sa_mask));
+		assert(!sigaddset(&sa.sa_mask, SIGSEGV));
+		assert(!sigaction(SIGSEGV, &sa, NULL));
+	}
+	{ /* SIGILL */
+		struct sigaction sa = { 0 };
+		sa.sa_flags = SA_SIGINFO | SA_RESTART;
+		sa.sa_sigaction = sig_h;
+		assert(!sigemptyset(&sa.sa_mask));
+		assert(!sigaddset(&sa.sa_mask, SIGSEGV));
+		assert(!sigaction(SIGILL, &sa, NULL));
+	}
+	{ /* SIGTRAP */
+		struct sigaction sa = { 0 };
+		sa.sa_flags = SA_SIGINFO | SA_RESTART;
+		sa.sa_sigaction = sig_h;
+		assert(!sigemptyset(&sa.sa_mask));
+		assert(!sigaddset(&sa.sa_mask, SIGTRAP));
+		assert(!sigaction(SIGTRAP, &sa, NULL));
+	}
+}
+
 __attribute__((constructor(0xffff))) static void __zpoline_init(void)
 {
 #ifdef SUPPLEMENTAL__REWRITTEN_ADDR_CHECK
@@ -550,6 +670,12 @@ __attribute__((constructor(0xffff))) static void __zpoline_init(void)
 			MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE,
 			-1, 0)) != MAP_FAILED);
 #endif
+	assert((replaced_syscall_list = mmap(NULL,
+			0x1000 * (((sizeof(struct replaced_syscall) * MAX_REPLACED_SYSCALL) / 0x1000) + ((sizeof(struct replaced_syscall) * MAX_REPLACED_SYSCALL) % 0x1000 ? 1 : 0)),
+			PROT_READ | PROT_WRITE | PROT_EXEC,
+			MAP_PRIVATE | MAP_ANONYMOUS,
+			-1, 0)) != MAP_FAILED);
+	setup_signal();
 	setup_trampoline();
 	rewrite_code();
 	load_hook_lib();

The patch can be applied by the following commands.
git clone https://github.com/yasukata/zpoline.git

cd zpoline

git checkout 0a349e65c102f8f9bdbbf6da0a52c4006589178b main.c

Let's say the patch above is saved in diff.patch.
patch main.c diff.patch

make

Then, we will have a binary libzpoline.so.
test program
The following is a simple test program.
#include <stdio.h>

extern long skip_syscall_0(void);
extern long skip_syscall_1(void);
extern long skip_syscall_2(void);
extern long skip_syscall_3(void);
extern long leaf_fn(void);

void __impl(void)
{
	asm volatile (
	".globl skip_syscall_0 \n\t"
	"skip_syscall_0: \n\t"
	"mov $5,%rax \n\t"
	"jmp skip0 \n\t"
	"syscall \n\t"
	"skip0: \n\t"
	"mov $100,%rax\n\t"
	"nop \n\t"
	"nop \n\t"
	"nop \n\t"
	"ret \n\t"
	);
	asm volatile (
	".globl skip_syscall_1 \n\t"
	"skip_syscall_1: \n\t"
	"mov $5,%rax \n\t"
	"jmp skip1 \n\t"
	"syscall \n\t"
	"nop \n\t"
	"skip1: \n\t"
	"mov $100,%rax\n\t"
	"nop \n\t"
	"ret \n\t"
	);
	asm volatile (
	".globl skip_syscall_2 \n\t"
	"skip_syscall_2: \n\t"
	"mov $5,%rax \n\t"
	"jmp skip2 \n\t"
	"syscall \n\t"
	"nop \n\t"
	"nop \n\t"
	"skip2: \n\t"
	"mov $100,%rax\n\t"
	"nop \n\t"
	"ret \n\t"
	);
	asm volatile (
	".globl skip_syscall_3 \n\t"
	"skip_syscall_3: \n\t"
	"mov $5,%rax \n\t"
	"jmp skip3 \n\t"
	"syscall \n\t"
	"nop \n\t"
	"nop \n\t"
	"nop \n\t"
	"skip3: \n\t"
	"mov $100,%rax\n\t"
	"nop \n\t"
	"ret \n\t"
	);
	asm volatile (
	".globl leaf_fn \n\t"
	"leaf_fn: \n\t"
	"movq $20, -8(%rsp) \n\t"
	"movq $39,%rax \n\t"
	"syscall \n\t"
	"nop \n\t"
	"nop \n\t"
	"movq -8(%rsp), %rax \n\t"
	"nop \n\t"
	"nop \n\t"
	"ret \n\t"
	);
}

int main(void)
{
	printf("0: %lu\n", skip_syscall_0());
	printf("1: %lu\n", skip_syscall_1());
	printf("2: %lu\n", skip_syscall_2());
	printf("3: %lu\n", skip_syscall_3());
	printf("%lu\n", leaf_fn());
	return 0;
}

If the test program above is saved in a file named program_above.c, the following command will generate an executable file a.out.
gcc program_above.c

output
The expected output from the test program looks like this.

the first 4 outputs (0: 100 ... 3: 100) check whether it can handle the execution at +2, +3, ...
the last line mimics a leaf function that returns a value located in the red zone, and supposedly we should have 20.

$ ./a.out
0: 100
1: 100
2: 100
3: 100
20

The output of version 0a349e65c102f8f9bdbbf6da0a52c4006589178b is as follows.
$ LD_PRELOAD=./libzpoline.so ./a.out
env LIBZPHOOK is empty, so skip to load a hook library
0: 100
1: 100
2: 100
3: 100
93908857225659

The following is the version having the patch above.
$ LD_PRELOAD=./libzpoline.so ./a.out
env LIBZPHOOK is empty, so skip to load a hook library
0: 100
1: 100
2: 100
3: 100
20

Here, the newer version seems to be able to handle +2, +3, +4 instructions and repair the value in the red zone by %rcx.
So, in summary, I think the direction of your approach is super nice.
The patch above is not perfect yet, and the following is the to-do list in my mind.

generally, the signal handling part has to be improved.
specifically, when the overwritten 6 bytes contain a piece of code that determines an address through the offset from %rip, the current implementation fails to point to the correct pointer; we need a bit of translation.
this approach is not compatible with the optimization for the code at address 0; I will explore another way to have the performance gain.
to maintain the mapping of the overwritten address and previously located program, I made a list of the replaced syscall instructions; the lookup time would negate the performance, so, I may need to use an efficient search algorithm.

I will update the code here when I have progress.
I really appreciate that you provided an excellent solution. Thank you very much.

Maybe.. add a pop %rcx before call *%rax and add 0x60; int3; int3 after it... This allow the fix of red zone using value in rcx.
The seq is pop %rcx; call *%rax; 0x60; int3; int3 = 0x59 0xff 0xd0 0x60 0xcc 0xcc.

Naive question about pop %rcx . Doesn't %rcx itself get clobbered then ? Shouldn't we need to save %rcx somewhere before pop %rcx ?

Maybe.. add a pop %rcx before call *%rax and add 0x60; int3; int3 after it... This allow the fix of red zone using value in rcx.
The seq is pop %rcx; call *%rax; 0x60; int3; int3 = 0x59 0xff 0xd0 0x60 0xcc 0xcc.

Naive question about pop %rcx . Doesn't %rcx itself get clobbered then ? Shouldn't we need to save %rcx somewhere before pop %rcx ?

System call will clobber rcx and r11 anyway, and they are not used to pass argument to the kernel, thus as long as it starts from the location of original syscall, it is fine.

Please let me report some updates.
I am sorry, but I am still working on the solution presented by @Yangff and have not finished it yet.
The current direction is that, during the initialization phase, I save the code, overwritten with 0x59 0xff 0xd0 0x60 0xcc 0xcc, on some other place and execute it just before the hook function returns; here, I need to convert the saved code that includes an instruction accessing data via an offset from the value of %rip so that it will access the appropriate address, and this conversion is a bit more complicated than I thought.
Essentially, I am still exploring a simple way to translate the saved code, and please let me continue the investigation.
Besides this, I made a temporary workaround that preserves the first 8 bytes of the red zone for our callq *%rax by rewriting the code accessing the red zone from N(%rsp) to N-8(%rsp) where N is a negative value.
The limitations of this workaround are:

Because we use 136 bytes as the red zone, the kernel may overwrite the exceeded 8 bytes if a process, executing a leaf function, is pre-empted.
We cannot apply this code replacement if N of N(%rsp) is smaller than -120 because the specification of -129 needs two bytes and the two-byte offset specification does not fit in the one-byte space having the original offset value.
The current implementation does not track the access to the red zone through a different register (e.g., mov %rsp,%rax; mov $1,N(%rax);).

Despite these limitations, I applied this workaround to the main branch because I found the issue reported in https://github.com/yasukata/zpoline/issues/1 has disappeared and I thought it was slightly better than nothing.
Anyway, I will continue the exploration.
Again, I deeply appreciate all of your kind help.

Hmmm.. just another thought.. is it possible to simply enable the last branch record to capture the return address instead of using call? Using jmp *%rax and restricting the max depth = 1, and limit the types to near jmp only, it should be possible to retrive the RIP using RDMSR after the NOPs?

@Yangff
Thank you very much for your idea.
I did not know that the CPUs have the Last Branch Record (LBR) feature, and it looks very useful.
While I am still looking through the CPU manual and relevant materials, please let me report my thoughts.

I'm not sure what could happen if there is CPU interrupt in between this... I assume the kernel should be able to keep the records purely for user events?

I guess, the MSR_LBR_SELECT field would allow us to configure the LBR feature to omit branch records while the CPU is in ring 0; so, I expect we can omit the branch records for interrupts.
I thought of the case where multiple processes run on the same CPU core; my concern was that the branch records in the LBR storage space of the CPU would be mixed (i.e., some branch records are for process A, and some are for process B). For this case, I am not sure yet, but the Linux kernel seems to implement a feature to preserve an LBR state for each process/thread at every process/thread switching ( https://github.com/torvalds/linux/commit/47125db27e47e9d44c878bf8925aa057824bb0d5 ) and it would allow us to get the branch records of a specific process even though there are multiple processes on the same CPU core. On the other hand, to activate this feature, we may need to do something, in user space, to communicate with the kernel-space perf subsystem through the perf_event_open system call interface; essentially, I need to get more familiar with this.
Anyway, I think that the approach using the LBR feature is feasible.
Please let me continue to work on your solution embedding 0x59 0xff 0xd0 0x60 0xcc 0xcc, and besides this, I will investigate the LBR-based approach.
I really appreciate that you provided a very cool idea along with useful information.

@Yangff
Thank you very much for your idea.
I did not know that the CPUs have the Last Branch Record (LBR) feature, and it looks very useful.
While I am still looking through the CPU manual and relevant materials, please let me report my thoughts.

I'm not sure what could happen if there is CPU interrupt in between this... I assume the kernel should be able to keep the records purely for user events?

I guess, the MSR_LBR_SELECT field would allow us to configure the LBR feature to omit branch records while the CPU is in ring 0; so, I expect we can omit the branch records for interrupts.
I thought of the case where multiple processes run on the same CPU core; my concern was that the branch records in the LBR storage space of the CPU would be mixed (i.e., some branch records are for process A, and some are for process B). For this case, I am not sure yet, but the Linux kernel seems to implement a feature to preserve an LBR state for each process/thread at every process/thread switching ( torvalds/linux@47125db ) and it would allow us to get the branch records of a specific process even though there are multiple processes on the same CPU core. On the other hand, to activate this feature, we may need to do something, in user space, to communicate with the kernel-space perf subsystem through the perf_event_open system call interface; essentially, I need to get more familiar with this.
Anyway, I think that the approach using the LBR feature is feasible.
Please let me continue to work on your solution embedding 0x59 0xff 0xd0 0x60 0xcc 0xcc, and besides this, I will investigate the LBR-based approach.
I really appreciate that you provided a very cool idea along with useful information.

yeah. The LBR is per CPU so the kernel needs to do something when switching process..
I think the biggest problem is there is no easy way to read this MSR.. it reuqires privileged instructions for both RDMSR or XSAVES in order to read..

@Yangff

I think the biggest problem is there is no easy way to read this MSR.. it reuqires privileged instructions for both RDMSR or XSAVES in order to read..

I see, but I guess executing one additional system call for RDMSR would be acceptable especially if we limit the access to the MSR only for the leaf function cases; so, the use of the LBR feature looks like a plausible option to me.
At least, you have expanded the option list for me; I really appreciate your kind help.

@pdlan
Thank you for your message.
The current implementation already has a quick check to detect red zones and try to avoid overwriting them https://github.com/yasukata/zpoline/issues/9#issuecomment-1855093788.
But, I think there is room for improvement particularly for the detection accuracy.
I expect your idea along with the use of one of the existing static analysis tools can improve the check mechanism. So, I put this on my to-do list.
Thank you very much for providing me with your idea and the pointer to the tool.


Missing 0.3.5 changelog and tag

ws4py 0.3.5 was released to PyPI 2016-06-05. However, the git repo doesn't have a tag for the commit that was released nor an updated changelog. Both would be useful when reviewing the changes before updating the Debian package. Thanks!

Damn. You're right. I totally forgot about those :(
I will update them.

The tag would be really useful -- after tagging, won't 0.3.5 show up on Github under releases?

The tag and changelog have been updated for 0.3.5, now we know where we are :)


Bug? node debug app.js not serving updated assets

Seems like when you run node debug app.js, and changes have been made in the assets folder like the javascript/css/html, the changes are not reflected on the site.
The assets only seems to recompile properly after a sails lift.
Running a restart in node debugger more doesn't do anything either.

Happens to me as well, here are a few more details form my dev env:
####Webstorm's output when I run debug (Won't update .tmp form assets [aka the problem]):
/usr/local/bin/node --debug-brk=58834 --nolazy /usr/local/lib/node_modules/sails/bin/sails.js lift
debugger listening on port 58834

info: Starting app...

info: debug:
--------------------------------------------------------info:
               .-..-.
info:
debug: :: Tue May 19 2015 23:29:15 GMT+0300 (IDT)
info:    Sails              <|    .-..-.
debug: Environment : development
info: debug: Port        : 1337
debug:    v0.11.0             |\
--------------------------------------------------------info:
                      /|.\
info:                      / || \
info:                    ,'  |'  \
info:                 .-'.-==|/_--'
info:                 `--'-------'
info:    __---___--___---___--___---___--___
info:  ____---___--___---___--___---___--___-__
info:
info: Server lifted in `/Users/alonamir/Work/9veg`
info: To see your app, visit http://localhost:1337
info: To shut down Sails, press <CTRL> + C at any time.

####Webstorm's output when I run debug (Updates .tmp from assets like it should):
/usr/local/bin/node /usr/local/lib/node_modules/sails/bin/sails.js lift

info: Starting app...

info:
debug: info:                .-..-.
--------------------------------------------------------info:
debug: :: Tue May 19 2015 23:31:27 GMT+0300 (IDT)

info: debug: Environment : development
   Sails              <|    .-..-.
info: debug: Port        : 1337
debug:    v0.11.0             |\
--------------------------------------------------------info:
                      /|.\
info:                      / || \
info:                    ,'  |'  \
info:                 .-'.-==|/_--'
info:                 `--'-------'
info:    __---___--___---___--___---___--___
info:  ____---___--___---___--___---___--___-__
info:
info: Server lifted in `/Users/alonamir/Work/9veg`
info: To see your app, visit http://localhost:1337
info: To shut down Sails, press <CTRL> + C at any time.

@aabluedragon I have the same issue
any updates on this?

I managed to get it working by removing any "--debug" or "--debug-brk" arguments from process.execArgv before the child process is created:
      var rememberArgs = [];

      for(var i = 0;i<process.execArgv.length; i++){
        if(process.execArgv[i].match(/--debug/)|| process.execArgv[i].match(/--debug-brk/)){
          rememberArgs.push(process.execArgv[i]);
          process.execArgv.splice(i, 1);
        }
      }

@https://github.com/balderdashy/sails/blob/master/lib/hooks/grunt/index.js#L59
and after it is created pushing them back:
      for(var j = 0; j<rememberArgs.length;j++){
        process.execArgv.push(rememberArgs[j]);
      }

@sgress454
may I do a PR for this?

@sgress454 nevermind... this issue will probably be fixed in 0.12 as stated by you in https://github.com/balderdashy/sails/issues/2670


Fetching url_picture property when searching for author by id

When searching for an author by name, the url_picture property is set.
>>> list(scholarly.scholarly.search_author('Sebastian Lapuschkin))[0]

{'affiliation': 'Postdoctoral Research Associate, Fraunhofer Heinrich Hertz '
                'Institute',
 'citedby': 3363,
 'email'<EMAIL_ADDRESS> 'filled': False,
 'id': 'wpLQuroAAAAJ',
 'interests': ['Interpretability',
               'Explainable AI',
               'Machine Learning',
               'Artificial Intelligence',
               'Deep Learning'],
 'name': 'Sebastian Lapuschkin (né Bach)',
 'url_picture': 'https://scholar.google.com/citations?view_op=medium_photo&user=wpLQuroAAAAJ'}

However, if the same author info is requested via their author ID, the url_picture property is not set. This might be great however, since name based resolution sometimes is not unambiguous.
>>> scholarly.scholarly.search_author_id('wpLQuroAAAAJ')

{'affiliation': 'Postdoctoral Research Associate, Fraunhofer Heinrich Hertz '
                'Institute',
 'filled': False,
 'id': 'wpLQuroAAAAJ',
 'interests': ['Interpretability',
               'Explainable AI',
               'Machine Learning',
               'Artificial Intelligence',
               'Deep Learning'],
 'name': 'Sebastian Lapuschkin (né Bach)'}

Is there any way to obtain the url_property field when searching authors by id

Issue probably is related to #132

Issue fixed

from scholarly import scholarly

author = next(scholarly.search_author('Sebastian Lapuschkin'))

scholarly.pprint(author)

author = scholarly.search_author_id('wpLQuroAAAAJ')

scholarly.pprint(author)

returns
{'affiliation': 'Postdoctoral Research Associate, Fraunhofer Heinrich Hertz '
                'Institute',
 'citedby': 3568,
 'email_domain'<EMAIL_ADDRESS> 'filled': False,
 'interests': ['Interpretability',
               'Explainable AI',
               'Machine Learning',
               'Artificial Intelligence',
               'Deep Learning'],
 'name': 'Sebastian Lapuschkin (né Bach)',
 'scholar_id': 'wpLQuroAAAAJ',
 'source': 'SEARCH_AUTHOR_SNIPPETS',
 'url_picture': 'https://scholar.google.com/citations?view_op=medium_photo&user=wpLQuroAAAAJ'}

{'affiliation': 'Postdoctoral Research Associate, Fraunhofer Heinrich Hertz '
                'Institute',
 'filled': False,
 'interests': ['Interpretability',
               'Explainable AI',
               'Machine Learning',
               'Artificial Intelligence',
               'Deep Learning'],
 'name': 'Sebastian Lapuschkin (né Bach)',
 'scholar_id': 'wpLQuroAAAAJ',
 'source': 'AUTHOR_PROFILE_PAGE',
 'url_picture': 'https://scholar.googleusercontent.com/citations?view_op=view_photo&user=wpLQuroAAAAJ&citpid=10'}

Thank you for the fix, this definitely is nice!
However, there is still deviating behavior between search_author and search_author_id: If the sought after author did not set a profile picture, the former returns a dict containing the field url_picture pointing towards the default author pic, while the latter does not contain the field. I can of course manually create the field and infer its content, but the behavior is inconsistent.
A bunch of other fields are missing as well, e.g. citedby, which is a weird thing to get when searching by ambiguous name instead of specific id.
Is there any chance to further align both  outputs in this respect? Code to reproduce below:
from scholarly import scholarly

author = next(scholarly.search_author('Djordje Slijepcevic'))

scholarly.pprint(author)

author = scholarly.search_author_id('faWNz2YAAAAJ')

scholarly.pprint(author)

Output:
{'affiliation': 'Researcher, Institute of Creative\\Media/Technologies, St. '
                'Pölten University of Applied',
 'citedby': 66,
 'email_domain'<EMAIL_ADDRESS> 'filled': False,
 'interests': ['Machine Learning',
               'Deep Learning',
               'Computer Vision',
               'Gait Analysis'],
 'name': 'Djordje Slijepcevic',
 'scholar_id': 'faWNz2YAAAAJ',
 'source': 'SEARCH_AUTHOR_SNIPPETS',
 'url_picture': 'https://scholar.google.com/citations?view_op=medium_photo&user=faWNz2YAAAAJ'}

{'affiliation': 'Researcher, Institute of Creative\\Media/Technologies, St. '
                'Pölten University of Applied',
 'filled': False,
 'interests': ['Machine Learning',
               'Deep Learning',
               'Computer Vision',
               'Gait Analysis'],
 'name': 'Djordje Slijepcevic',
 'scholar_id': 'faWNz2YAAAAJ',
 'source': 'AUTHOR_PROFILE_PAGE'}


[SPARK-32180][PYSPARK][DOCS] Getting started-Installation guide for pyspark doc

What changes were proposed in this pull request?
This PR proposes to add getting started- installation to new PySpark docs.
Why are the changes needed?
Better documentation.
Does this PR introduce any user-facing change?
No. Documentation only.
How was this patch tested?
Generating documents locally.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Hi @HyukjinKwon,
I was not sure how to add you as a Reviewer for this Pull request, thus adding this comment. I would like to mention a few points:

Baseline description: I have used this pull request as a reference since this is my first pull request- #29385
or https://github.com/apache/spark/pull/29385.
Most of the information used for the "Installation Page" has come from Koalas documentation-https://koalas.readthedocs.io/en/latest/getting_started/install.html. Am I supposed to mention this as reference/credit anywhere in this documentation?
Personally I have used only PyPI as an installation mechanism thus haven't tried the other three. A prerequisite for PyPI installation was the availability of JAVA 8 path in JAVA_HOME environment variable. Please let me know if that's something I need to add explicitly in the "Dependencies" section.
Where ever necessary I have updated the links.

ok to test

Test build #127366 has started for PR 29410 at commit e93b4e1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31986/
Test PASSed.

Test build #127366 has finished for PR 29410 at commit e93b4e1.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127366/
Test FAILed.

Test build #127465 has started for PR 29410 at commit 5df88b4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32086/
Test PASSed.

Test build #127465 has finished for PR 29410 at commit 5df88b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127465/
Test PASSed.

@HyukjinKwon,
Thanks for your helpful comment. I have done the suggested changes but if you still find something which requires modification, please let me know, I will update it.
Thanks!!!

Personally I have used only PyPI as an installation mechanism thus haven't tried the other three. A prerequisite for PyPI installation was the availability of JAVA 8 path in JAVA_HOME environment variable. Please let me know if that's something I need to add explicitly in the "Dependencies" section.

This is a good point ..  we should mention Java is required. Currently, yes, Java 8 should be available in the system.

Most of the information used for the "Installation Page" has come from Koalas documentation-https://koalas.readthedocs.io/en/latest/getting_started/install.html. Am I supposed to mention this as reference/credit anywhere in this documentation?

I think no need. But I would like to say that Koalas installation page is not good enough .. I wrote all most of documentations by myself alone in Koalas so it's not complete yet .. 😢

Test build #128147 has started for PR 29410 at commit 4284fb4.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32772/
Test PASSed.

Test build #128147 has finished for PR 29410 at commit 4284fb4.

This patch fails Python style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128147/
Test FAILed.

Test build #128148 has started for PR 29410 at commit 30cbdfa.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32773/
Test PASSed.

@HyukjinKwon ,
Apology for the delay in updating the required changes. I have made the suggested changes and I pushed it for your review.
I can see a conflict test failed and trying to understand how to fix it.

Test build #128148 has finished for PR 29410 at commit 30cbdfa.

This patch fails MiMa tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128148/
Test FAILed.

@rohitmishr1484, thanks. Looks getting there. Can you resolve the conflicts? See also "The Review Process" at https://spark.apache.org/contributing.html to make this PR synced to the latest master.

Test build #128258 has started for PR 29410 at commit 001814f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32886/
Test PASSed.

Oops, looks something went wrong during syncing. Feel free to close this and create new PR :-)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128258/
Test PASSed.


Windows worker cannot connect after upgrading to 3.14.1

We have been using concourse binary for windows worker. After upgrading from concourse 3.9.2 to 3.14.1, concourse workers cannot connect to TSA
“worker”,“message”:“worker.setup.no-assets”,“log_level”:1,“data”:{“session”:“1”}}
{“timestamp”:“1528724301.973331690”,“source”:“worker”,“message”:“worker.sweep-starting”,“log_level”:1,“data”:{}}
{“timestamp”:“1528724301.973331690”,“source”:“worker”,“message”:“worker.reaper.reaper-server.started-reaper-process”,“log_level”:1,“data”:{“garden-addr”:“<IP_ADDRESS>:7777”,“server-port”:“7799”,“session”:“5.1”}}
{“timestamp”:“1528724301.973331690”,“source”:“worker”,“message”:“worker.beacon.beacon.beacon-client.failed-to-connect-to-tsa”,“log_level”:2,“data”:{“error”:“dial tcp: address : missing port in address”,“session”:“4.1.1”}}
{“timestamp”:“1528724301.973331690”,“source”:“worker”,“message”:“worker.beacon.restarting”,“log_level”:2,“data”:{“error”:“failed to connect to TSA”,“session”:“4”}}
{“timestamp”:“1528724301.974332094”,“source”:“worker”,“message”:“worker.garden.started”,“log_level”:1,“data”:{“session”:“2”}}
We have been starting the concourse worker using cmd:
d:\concourse\concourse.exe worker /work-dir d:\concourse\work /tsa-host /tsa-public-key d:\concourse\keys\tsa_host_key.pub /tsa-worker-private-key d:\concourse\keys\worker_key >> d:\concourse\logs\worker-log.txt
We use BOSH deployment for Concourse web and Linux workers.
Is there any configuration change that needs to be done?

Was missing port when starting windows worker


GC: wasmparser: Subtypes

Introducing the SubType struct:
pub struct SubType {
    pub is_final: bool,
    pub supertype_idxs: Vec<u32>,
    pub structural_type: StructuralType,
}

Here, StructuralType is an enum (previously named Type).
supertype_idxs is a vector of indexes of supertypes, limited to at most one supertype as per GC MVP.
A valid subtype matches its supertypes. Supertypes cannot be final.

x86_64-linux and x86_64-windows CI jobs were canceled for some reason.

@alexcrichton I believe I've addressed all @fitzgen's feedback (thank you!).

@alexcrichton I believe I've addressed all your comments (so far).


Multiple word US format failing

Running curl 'wttr.in/~San+Francisco?format="%t"' gets me
Unknown location; please try ~37.7792808,-122.4192362

@dessalines Actually, it is not the real problem; the real problem is that today our datasource is experiencing partial outage, and it does not deliver information data for almost all locations in the world. I don't know how much does it take to recover, but certainly it will not happen today.

What's the data source? It might also be a good idea for that to pull that as a static file, that is periodically pulled on every new build of wttr. Especially if it happens frequently.

I am not sure that I understand what you mean, what kind of file should be pulled.
Anyway, the problem is fixed, please test

Its good, thanks.

what kind of file should be pulled.

What's your data source for locations? You could save a csv of the cached location names, if those go down frequently.

@dessalines No, you've got me wrong. I meant not the location data source, it is not a problem at all (though even this can't be converted into a CSV file because we support all possible languages and all possible location names), but the problem was with the weather data source; the error message was just irritating.

Ah okay, then maybe a better error message returned is necessary, since that wasn't the actual error. Something like weather service down / unavailable

Yes, you are right. By the way, after you opened the issue, I changed the error message to the default  one (that we have usually in such cases).


Advfilter needs more unit test cases

Advfilter is a relatively new feature. Needs more testing.

New test cases written


[Improvement]: Integrate a subset of semtypes into jBallerina runtime

Description
$subject.

This is to be done similar to #41066 (ie. initially we will support a state where part of the type is still a BType and rest is SemType). For this I have considered 2 approaches,

At runtime split a given BType to a SemType component as well as a BType component and type check them separately.
Make the SemType the "top level" type and make the BType another sub type data.

First approach is not removing any of the BTypes and instead we are creating additional SemTypes (and extra BType for the BType component). This means we have a performance overhead (in nBallerina I measured ~15% slowdown mainly due to extra instance creation). Second approach should not have that weakness since we are replacing BTypes with SemTypes. Currently I am exploring this approach.

I am planning to convert runtime type representation to SemTypes in fallowing stages

[ ] Convert the runtime representation of  nil, int, boolean, string, float, decimal, handle, never and their sub types to use SemTypes.
[ ] Convert the runtime representation of list, map and xml types to SemTypes.
[ ] Convert the runtime representation of function type to use SemTypes

At any stage unions and intersections will represent the parts that are supported under that stage as SemTypes as well. Note that list, map and xml types will require porting the cell types (used to represent mutability) as well. Also note all other types such as table are represented using combinations of above types.

Ballerina Type
Underlying SemType

table
list + mapping

error, object
map

This is to be done similar to #41066 (ie. initially we will support a state where part of the type is still a BType and rest is SemType). For this I have considered 2 approaches,
1. At runtime split a given `BType` to a `SemType` component as well as a `BType` component and type check them separately.

2. Make the` SemType` the "top level" type and make the `BType` another sub type data.

First approach is not removing any of the BTypes and instead we are creating additional SemTypes (and extra BType for the BType component). This means we have a performance overhead (in nBallerina I measured ~15% slowdown mainly due to extra instance creation). Second approach should not have that weakness since we are replacing BTypes with SemTypes. Currently I am exploring this approach.

Extension of option 2 is to make existing BTypes extends SemType. Basically for each type supported by the runtime API their corresponding implementations will implement both the SemType as well as that interface. Type operations are still immutable and still result in new Semtypes that will only implement the base Type interface. But this will allow things like standard library to continue using (most of) the types as is with minimum modifications. Internally these new types will just become adapters between semtype API and current type APIs

Closing with #43279


Currency input?

Describe the feature
Do you support currency inputs like  vue-currency-inputÏ? If not, are they compatible with formkit?
Thank you!

You can wrap them as FormKit inputs. Eventually inputs like the currency input will be available as part of FormKit Pro. You are also welcome to create a plugin and share it with the FormKit comunity.

@684efs3 did you achieved to make it work, my current solution is kind of flaky.
<template>
  <input ref="inputRef" class="formkit-input" :value="context._value" @input="handleInput"/>
</template>

<script setup lang="ts">
import {CurrencyInputOptions, useCurrencyInput} from "vue-currency-input";
import {PropType} from "@vue/runtime-core";

const props = defineProps({
  modelValue: Number,
  options: {
    type: Object as PropType<CurrencyInputOptions>,
    default: () => ({
      currency: 'EUR',
      hideGroupingSeparatorOnFocus: false,
      precision: 2
    })
  },
  context: Object
})

const {inputRef} = useCurrencyInput(props.options)

function handleInput(e) {
  props.context.node.input(e.target.value)
}
</script>

``

As an example, when you set a default price, let's say 1000000, what I expect to show it like: 1.000.000,00 but it is represented 1000000, when I focus the input it will changes to 1.000.000,00.


Add dev tool to rename files

Note there are many other - shorter - ways to accomplish this same objective using the shell, but it can be a good exercise to find ways to do things with more than one tool.


Why doesn't godoc.org look like docs in golang.org/pkg?

 Perhaps I'm not alone.

Are you talking about the style/presentation differences?

To answer the question, they're different because they're different projects. gddo was originally created by Gary Burd roughly in 2013, see 68142c388cce0bac9fbd83c49ec8a431b25a2470. I guess it was not a goal to have identical look and feel compared to golang.org/pkg. I presume the changes were perceived as improvements over golang.org/pkg.
Which one looks better? I don't know, it probably depends on probably personal preference and experience. I don't mind either, but I use godoc.org 95%+ of the time and golang.org/pkg almost never (why use it, aside from looking at APIs in beta Go versions; it doesn't support 3rd party packages and doesn't have 'F' jump navigation shortcut).
Should godoc.org be changed to look identically to golang.org/pkg? I don't know. It's not my call. But probably not worth it, and not desired.

Oh yeah, I'm talking about the style difference. It's just my personal taste and I know that it's not a big deal, but the golang.org/pkg style seems much more consistent to me than godoc.org style, which just reminds me of a poorly used bootstrap template with inconsistent spacing and fonts.
But you know, if no one else feels like I do, no big deal.

Personally, I'd like it if there was only one (best) style. But I doubt it's going to happen, and I don't feel very strongly about it.

Personally I mostly use godoc -http=:6060 and so the golang.org-like local rendering. And it works not just for stdlib but also for any third party packages that I have in $GOPATH/src.


An error dialog pops up when opening Storage Explorer on MacOS

Storage Explorer Version: 1.32.0-dev (95)
Build Number: 20231014.5
Branch: rel/1.32.0
Platform/OS: MacOS Sonoma 14.0 (Apple M1 Pro)
Architecture: arm64
How Found: From running test cases
Regression From: Previous release (1.31.2)
Steps to Reproduce

Download the Mac build StorgeExplorer-darwin-arm64.zip -> Expand the .zip file.
Double click the file 'Microsoft Azure Storage Explorer' to open Storage Explorer.
Check no error dialog pops up.

Expected Experience
No error dialog pops up.
Actual Experience
An error dialog pops up.

Additional Context
This issue doesn't reproduce for x64 architecture.

Verified this issue on the build 20231018.1. Fixed.


Applications are Unavailable - Authentication issues

We are seeing widespread authentication issues and are working as quickly as we can to correct it.

We've identified the issue with authentication, and have begun the recovery process.

Resolved: Desk is back up in cbcc165.

Post Mortem
Cause: Our background logging service saw reduced connectivity overnight due to uncontrolled network conditions.
Effects: This caused a backup of messages in our message queue cluster, which in turn created a resource contention issue with TCP sockets around 8:10 am MDT. Eventually, widespread connection issues arose with the rest of the services using our messaging stack, causing requests to fail.
Action Taken: A manual reconnection of our logging service, which worked through the backed-up queue and relieved resources for further connections to succeed.
Root Issue & Moving Forward: Failure of the logging service to reconnect and resume operations was the cause of the issue, and we are working as fast as possible to ensure it doesn't happen again. We will be adding forewarning alerts and monitoring in the future so we can further mitigate issues of this nature.


CDIBuilder

In the Vaadin 8 variant I could work with Dependency Injection:
AppLayout.getCDIBuilder(behaviour).withViewProvider(() -> viewProvider)
How to do this in the Vaadin 10 variant?

Vaadin Spring support is currently not given. Mostly due to a lack of time from my side.

@mirkoliver I think there is no longer a ViewProvider.


Floaty only respond to touch for the first time on simulator when built by Xcode 14.

It only works for the first touch, then subsequent clicks are ignored. Tested on iOS 14 and iOS 16 simulators.
Works fine on real device tho.

i have same issue

i have same issue too

Facing same issue did anyone solved it successfully?


Add flag to be greedy or not with found slashing offences

We should decide the default behavior and expose a flag to change it

unstale


Compilation errors

Hi There,
I downloaded the v2.0.1 release and attempted to compile it. I am getting a bunch of compilation errors reproduced below.
$ mvn clean install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Azure Cosmos DB SQL API
[INFO] Async SDK for SQL API of Azure Cosmos DB Service
[INFO] Async SDK for SQL API of Azure Cosmos DB Service - Examples
[INFO] Async SDK for SQL API of Azure Cosmos DB Service - Benchmarking tool
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Azure Cosmos DB SQL API 2.0.1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ azure-cosmosdb-parent ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ azure-cosmosdb-parent ---
[INFO] Installing /Users/dbhati2/Downloads/azure-cosmosdb-java-2.0.1/pom.xml to /Users/dbhati2/.m2/repository/com/microsoft/azure/azure-cosmosdb-parent/2.0.1/azure-cosmosdb-parent-2.0.1.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Async SDK for SQL API of Azure Cosmos DB Service 2.0.1
[INFO] ------------------------------------------------------------------------
Downloading: http://xxx/content/groups/public/org/apache/maven/plugins/maven-javadoc-plugin/2.9.1/maven-javadoc-plugin-2.9.1.pom
Downloaded: http://xxx/content/groups/public/org/apache/maven/plugins/maven-javadoc-plugin/2.9.1/maven-javadoc-plugin-2.9.1.pom (16 KB at 40.0 KB/sec)
Downloading: http://xxx/content/groups/public/org/apache/maven/plugins/maven-javadoc-plugin/2.9.1/maven-javadoc-plugin-2.9.1.jar
Downloaded: http://xxx/content/groups/public/org/apache/maven/plugins/maven-javadoc-plugin/2.9.1/maven-javadoc-plugin-2.9.1.jar (360 KB at 194.0 KB/sec)
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ azure-cosmosdb ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ azure-cosmosdb ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/dbhati2/Downloads/azure-cosmosdb-java-2.0.1/sdk/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.6.0:compile (default-compile) @ azure-cosmosdb ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 184 source files to /Users/dbhati2/Downloads/azure-cosmosdb-java-2.0.1/sdk/target/classes
[INFO] /Users/dbhati2/Downloads/azure-cosmosdb-java-2.0.1/sdk/src/main/java/com/microsoft/azure/cosmosdb/JsonSerializable.java: Some input files use unchecked or unsafe operations.
[INFO] /Users/dbhati2/Downloads/azure-cosmosdb-java-2.0.1/sdk/src/main/java/com/microsoft/azure/cosmosdb/JsonSerializable.java: Recompile with -Xlint:unchecked for details.
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /Users/dbhati2/Downloads/azure-cosmosdb-java-2.0.1/sdk/src/main/java/com/microsoft/azure/cosmosdb/rx/internal/caches/AsyncCache.java:[93,60] no suitable method found for flatMap((vaule)->{...; },(err)->{ l...; },()->Observ[...]pty())
method rx.Observable.flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends R>>) is not applicable
(cannot infer type-variable(s) R
(actual and formal argument lists differ in length))
method rx.Observable.flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends R>>,int) is not applicable
(cannot infer type-variable(s) R
(actual and formal argument lists differ in length))
method rx.Observable.flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends R>>,rx.functions.Func1<? super java.lang.Throwable,? extends rx.Observable<? extends R>>,rx.functions.Func0<? extends rx.Observable<? extends R>>) is not applicable
(no instance(s) of type variable(s) T exist so that rx.Observable conforms to ? extends rx.Observable<? extends R>)
method rx.Observable.flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends R>>,rx.functions.Func1<? super java.lang.Throwable,? extends rx.Observable<? extends R>>,rx.functions.Func0<? extends rx.Observable<? extends R>>,int) is not applicable
(cannot infer type-variable(s) R
(actual and formal argument lists differ in length))
method rx.Observable.<U,R>flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends U>>,rx.functions.Func2<? super TValue,? super U,? extends R>) is not applicable
(cannot infer type-variable(s) U,R
(actual and formal argument lists differ in length))
method rx.Observable.<U,R>flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends U>>,rx.functions.Func2<? super TValue,? super U,? extends R>,int) is not applicable
(cannot infer type-variable(s) U,R
(argument mismatch; int is not a functional interface))
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Azure Cosmos DB SQL API ............................ SUCCESS [  0.208 s]
[INFO] Async SDK for SQL API of Azure Cosmos DB Service ... FAILURE [  4.723 s]
[INFO] Async SDK for SQL API of Azure Cosmos DB Service - Examples SKIPPED
[INFO] Async SDK for SQL API of Azure Cosmos DB Service - Benchmarking tool SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.026 s
[INFO] Finished at: 2018-08-31T13:40:30-07:00
[INFO] Final Memory: 20M/116M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.0:compile (default-compile) on project azure-cosmosdb: Compilation failure
[ERROR] /Users/dbhati2/Downloads/azure-cosmosdb-java-2.0.1/sdk/src/main/java/com/microsoft/azure/cosmosdb/rx/internal/caches/AsyncCache.java:[93,60] no suitable method found for flatMap((vaule)->{...; },(err)->{ l...; },()->Observ[...]pty())
[ERROR] method rx.Observable.flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends R>>) is not applicable
[ERROR] (cannot infer type-variable(s) R
[ERROR] (actual and formal argument lists differ in length))
[ERROR] method rx.Observable.flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends R>>,int) is not applicable
[ERROR] (cannot infer type-variable(s) R
[ERROR] (actual and formal argument lists differ in length))
[ERROR] method rx.Observable.flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends R>>,rx.functions.Func1<? super java.lang.Throwable,? extends rx.Observable<? extends R>>,rx.functions.Func0<? extends rx.Observable<? extends R>>) is not applicable
[ERROR] (no instance(s) of type variable(s) T exist so that rx.Observable conforms to ? extends rx.Observable<? extends R>)
[ERROR] method rx.Observable.flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends R>>,rx.functions.Func1<? super java.lang.Throwable,? extends rx.Observable<? extends R>>,rx.functions.Func0<? extends rx.Observable<? extends R>>,int) is not applicable
[ERROR] (cannot infer type-variable(s) R
[ERROR] (actual and formal argument lists differ in length))
[ERROR] method rx.Observable.<U,R>flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends U>>,rx.functions.Func2<? super TValue,? super U,? extends R>) is not applicable
[ERROR] (cannot infer type-variable(s) U,R
[ERROR] (actual and formal argument lists differ in length))
[ERROR] method rx.Observable.<U,R>flatMap(rx.functions.Func1<? super TValue,? extends rx.Observable<? extends U>>,rx.functions.Func2<? super TValue,? super U,? extends R>,int) is not applicable
[ERROR] (cannot infer type-variable(s) U,R
[ERROR] (argument mismatch; int is not a functional interface))
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :azure-cosmosdb

@deepub
This package is already published to central maven repo (https://mvnrepository.com/artifact/com.microsoft.azure/azure-cosmosdb/2.0.1). For using the sdk in your app you won't need rebuilding. Rebuilding would only be required if you are planning to make changes to the SDK or send a PR.
If you are planning to just use the sdk add the following dependency to your maven pom file:
<dependency>
  <groupId>com.microsoft.azure</groupId>
  <artifactId>azure-cosmosdb</artifactId>
  <version>2.0.1</version>
</dependency>

If you are planning to build this package yourself:
The error you are getting is a compilation error. Seems you are trying to build this on mac os x. On Mac OS X my java version is "1.8.0_181" and building works fine.
Please make sure your java version is at least 1.8.0_181 and try again.
Please let me know how that goes.


Add document, document fragment, document type

kitchen sink at this point, pulled from various drafts.
Ok, whittling down to keys which match the following criteria:

superkeys like api.Document
keys which refer directly to the document itself or change it, eg document.lastModified and `document.
document-level collections, eg document.images and document.forms (yes, there may be "primary" features for the thing being collected but the document shortcut is document-level)

Need to determine whether to put other interface impls in the feature or in the interface - eg, document.append is impl of Element... so do we add all the append keys to their feature or add them all to an Element feature? TBD will ask in chat/meeting.
Will see what's left after that culling and classify the remainders.

Ready for a first pass. There's not any real standalone feature candidates in the key list for Document. There's a couple notes on Caniuse keys that are related.
Open question is what approach for keys like createTreeWalker - there'll be a TreeWalker feature coming out of the DOM PR I've got going. Things like Ranges also will be feature. I assumed it'd go with the specific feature vs leaving it here, but worth asking if there's different opinion there.

I think a lot of this stuff is going to get swept into a monolithic DOM feature. I'd appreciate if other reviewers also had a look through https://github.com/web-platform-dx/web-features/pull/2007/files#r1813040012.

Closing to fold into #2007


Pimp up bug bounty process

Expand texts on incentivized testing and drive people to https://discord.gg/M3zh3VEAfm to help us test Kodadot with better and quality reports what doesn't work
Can be part of contributing.md or bugbounty.md as well so it's straight forward linked

Continue to https://docs.kodadot.xyz/incentives/incentivized-bug-reporting.html#rewards


prepareCommitRevealConfig(this.options.opType, fundingKeypair, new AtomicalsPayload(copiedData))

When I was in mint arc20, I transferred the btc to the address prompted by terminal, but this address is not in my wallet.json, how should I get the private key for this address.
I looked at the code and found that this address is generated by this code
const baseCommit: { scriptP2TR, hashLockP2TR } = prepareCommitRevealConfig(this.options.opType, fundingKeypair, new AtomicalsPayload(copiedData))
Where baseCommit.scriptP2TR.address is the address that appears in my terminal, how do I get the private key for this address?

I didn't have this problem a few days ago, but now I'm just like you.


PR #37 seems to introduce a regression that breaks wireguard installation

Installation with current master:
2018/03/02 16:25:21 Creating new cluster staging1 with 1 master(s), 1 worker(s), HA: false
2018/03/02 16:25:21 creating server 'staging1-master-01'...
  10s [====================================================================] 100%
2018/03/02 16:25:38 Created node 'staging1-master-01' with IP <redacted>
2018/03/02 16:25:38 creating server 'staging1-worker-01'...
  15s [====================================================================] 100%
2018/03/02 16:25:59 Created node 'staging1-worker-01' with IP <redacted>
2018/03/02 16:25:59 sleep for 30s...
staging1-master-01   : configure wireguard                 5.0% [--------------]
staging1-worker-01   : configure wireguard                27.3% [==>-----------]
2018/03/02 16:28:54 Failed to execute operation: No such file or directory

2018/03/02 16:28:54 > systemctl enable wg-quick@wg0 && systemctl restart wg-quick@wg0
2018/03/02 16:28:54
2018/03/02 16:28:54
2018/03/02 16:28:54 Run failed:Process exited with status 1

systemd log from the worker:
Mar 02 16:43:23 staging1-worker-01 wg-quick[9275]: Warning: `/etc/wireguard/wg0.conf' is world accessible
Mar 02 16:43:23 staging1-worker-01 wg-quick[9275]: [#] ip link add wg0 type wireguard
Mar 02 16:43:23 staging1-worker-01 kernel: wireguard: module verification failed: signature and/or required key missing - tainting kernel
Mar 02 16:43:23 staging1-worker-01 kernel: wireguard: WireGuard 0.0.20180218 loaded. See www.wireguard.com for information.
Mar 02 16:43:23 staging1-worker-01 kernel: wireguard: Copyright (C) 2015-2018 Jason A. Donenfeld<EMAIL_ADDRESS>All Rights Reserved.
Mar 02 16:43:23 staging1-worker-01 wg-quick[9275]: [#] wg setconf wg0 /dev/fd/63
Mar 02 16:43:23 staging1-worker-01 wg-quick[9275]: Line unrecognized: `PrivateKey='
Mar 02 16:43:23 staging1-worker-01 wg-quick[9275]: Configuration parsing error
Mar 02 16:43:23 staging1-worker-01 wg-quick[9275]: [#] ip link delete dev wg0
Mar 02 16:43:23 staging1-worker-01 systemd[1]<EMAIL_ADDRESS>Main process exited, code=exited, status=1/FAILURE
Mar 02 16:43:23 staging1-worker-01 systemd[1]: Failed to start WireGuard via wg-quick(8) for wg0.
-- Subject: Unit<EMAIL_ADDRESS>has failed

On the master, systemctl enable wg-quick@wg0 && systemctl restart wg-quick@wg0 cannot be executed.
Looking at the config files on worker (/etc/wireguard/wg0) and master (/etc/wireguard), it looks like neither PrivateKey nor PublicKey are set.

I get this error even after downgrading to 0.21 and 0.20

Just pushed a quick fix in my PR #40, you can use my branch as a quick fix while waiting for @xetys maybe to come up with a better fix than mine :)

Or just adding this https://github.com/xetys/hetzner-kube/pull/40/commits/300e04ef3eb4a030fd8b9037b5d585ff9ca05ee1#diff-69296e960e0cee640864ddc03d0d562aR91 should fix in your banches

tested manually. Sry for that little break


Explicitly calls for slope file in degrees

I was using a slope raster generated form outside WhiteboxTools and wasn't sure if my slope raster in percentage was appropriate. I didn't find the info in the tool's help so I had to go look the code. I could have also tried rasters in degrees, radians and percentage and looked at what made sense but... I didn't.

This is good. I should probably update the tool documentation as well.


Listen for some hotkeys on form fields

Thanks for the great lib!
Currently in the keyDownHandler I see this:
if (event.target instanceof Node && isFormField(event.target)) return

In my use-case I would like to attach an "Escape" hotkey to a cancel button, and a "Command+Enter" hotkey to a submit button, and have these hotkeys be active whilst the focus is in an input/textarea. The above line prevents this.
Understandable that single character and Shift+* hotkeys shouldn't be active, but what about everything else? What's the recommended approach here?

Hey @tobyzerner thanks for the issue!
This library explicitly ignores shortcuts fired in form fields because it can be very problematic with regard to accessibility.
If you need to enable those kind of shortcuts, it's recommended to bind an event listener directly to handle that specific use case. We do this at GitHub; hotkey is used as a convenience layer but cases that need more fine tuning are done manually.

Hey @tobyzerner thanks for the issue!
This library explicitly ignores shortcuts fired in form fields because it can be very problematic with regard to accessibility.
If you need to enable those kind of shortcuts, it's recommended to bind an event listener directly to handle that specific use case. We do this at GitHub; hotkey is used as a convenience layer but cases that need more fine tuning are done manually.

@keithamus How is this done with the command palette in github and listening to meta+k on all inputs across github?

https://github.com/github/hotkey/discussions/85


Disconnecting after X minutes of inactivity.

Is your feature request related to a problem? Please describe.
I love the bot, but when I forget to tell it to exit, it stays on, using up bandwidth.
I would love for the bot to automatically disconnect if there is no activity for a certain period of time, in order to save data and processing. Especially if people end up inviting the bot to other servers, I don't want the bot to be in the channel alone.
Describe the solution you'd like
Maybe the problem I'm describing is already fixed since I haven't waited long enough, but an option in the .env file for a disconnect time would be great. Possibly by checking how long the queue has been empty for, and after the time in the env is exceeded, disconnect automatically.

Its very good idea BUMP


I wish /docs/user-guide/volumes/ gave an example for hostPath

I wish /docs/user-guide/volumes/ gave an example for hostPath
The text implies you can do
hostPath: /path/mydir
But in fact it needs to be
hostPath:
path: /path/mydir

I've created a PR to fix this.


🐜  Detect non-null 'options.ignored'

The value of options.ignored can be a string, function, regexp, or mixed array.
Strangely, no tests failed before or after this change.
Probably need to add an edge case.

Ack, looks like the checkIgnored() call handles the other cases.
Now I understand that checkIgnored(stats) is only necessary when options.ignored is a function.
But I wonder... what if an array of ignore functions are used?

Coverage increased (+0.2%) to 98.6% when pulling b9c0db365dfc2e9137207024dde778b118a1cdd8 on aleclarson:fsevents-ignored-fix into 3f7f113ac67638d8e4b4956f770446b1d727c4df on paulmillr:master.

Coverage increased (+0.2%) to 98.6% when pulling b9c0db365dfc2e9137207024dde778b118a1cdd8 on aleclarson:fsevents-ignored-fix into 3f7f113ac67638d8e4b4956f770446b1d727c4df on paulmillr:master.

Coverage increased (+0.2%) to 98.6% when pulling b9c0db365dfc2e9137207024dde778b118a1cdd8 on aleclarson:fsevents-ignored-fix into 3f7f113ac67638d8e4b4956f770446b1d727c4df on paulmillr:master.

Closing for new architecture.


Key Vault virtual machine extension for Windows

In the document "Key Vault virtual machine extension for Windows" at the header Azure PowerShell deployment the example for version 3.0 is:
Build settings
$settings = ".\settings.json"
$extName =  "KeyVaultForWindows"
$extPublisher = "Microsoft.Azure.KeyVault"
$extType = "KeyVaultForWindows"

# Start the deployment
Set-AzVmExtension -TypeHandlerVersion "3.0" -ResourceGroupName <ResourceGroupName> -Location <Location> -VMName <VMName> -Name $extName -Publisher $extPublisher -Type $extType -SettingString $settings

However the parameter SettingString needs the config as a string, not the file containing the config. A possible solution is:
Build settings
$settings = (get-content -raw ".\settings.json")
$extName =  "KeyVaultForWindows"
$extPublisher = "Microsoft.Azure.KeyVault"
$extType = "KeyVaultForWindows"

# Start the deployment
Set-AzVmExtension -TypeHandlerVersion "3.0" -ResourceGroupName <ResourceGroupName> -Location <Location> -VMName <VMName> -Name $extName -Publisher $extPublisher -Type $extType -SettingString $settings

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: a52b0724-6b3b-0daf-7749-d18104d6a40b
Version Independent ID: 56e5c7c4-f456-fc2e-99da-2771eb162947
Content: Azure Key Vault VM Extension for Windows - Azure Virtual Machines
Content Source: articles/virtual-machines/extensions/key-vault-windows.md
Service: virtual-machines
Sub-service: extensions
GitHub Login: @msmbaldwin
Microsoft Alias: mbaldwin

@bmemvandenhoek
Thanks for your feedback! We will investigate and update as appropriate.

@bmemvandenhoek
Thanks for reporting this! We have created a PR for this issue and the changes should go live soon.


Fix ArrayType.equalTo to work with comparable non-orderable types

Attempt to fix #3542. Still having some trouble passing the new test, because it's not clear how comparable non-orderable types should be handled at lower layers.

Let me know when you address the comments.

@cberner This pull request seems good to me. Please take a look and then assign back to me.
@raghavsethi squash your commits.

looks good


Make WSL suggestion more reluctant

Overview
Addresses https://github.com/pilosa/pilosa/issues/2129

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Alan Bernstein seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Submit relation between Fin and ⊎ to stdlib

This is present in both Object.Power and in pi-dual. Should be in neither!

Object.Power no longer exists. But this is still in Functor.Power.
unpack is Data.Fin.splitAt.  The opposite function is apparently still not in Data.Fin !

Object.Power no longer exists. But this is still in Functor.Power.
unpack is Data.Fin.splitAt.  The opposite function is apparently still not in Data.Fin !

We actually have it twice!
https://github.com/agda/agda-stdlib/blob/f9a53e8699259c8567d374a31c2c5636efaf50d0/src/Data/Fin/Base.agda#L209
https://github.com/agda/agda-stdlib/blob/f9a53e8699259c8567d374a31c2c5636efaf50d0/src/Data/Fin/Permutation/Components.agda#L40

We actually have it twice!
https://github.com/agda/agda-stdlib/blob/f9a53e8699259c8567d374a31c2c5636efaf50d0/src/Data/Fin/Base.agda#L209
https://github.com/agda/agda-stdlib/blob/f9a53e8699259c8567d374a31c2c5636efaf50d0/src/Data/Fin/Permutation/Components.agda#L40

Two things @MatthewDaggitt :

probably there should an issue filed to remove that duplication?
what I meant was the inverse function to splitAt, defined inline in https://github.com/agda/agda-stdlib/blob/f9a53e8699259c8567d374a31c2c5636efaf50d0/src/Data/Fin/Properties.agda#L484-L502 in the "proof" that Fin (m + n) ≃ Fin m ⊎ Fin n (but which doesn't exist itself anywhere).  i.e.

unsplit :: {m n : ℕ} (i : ℕ) → Fin m ⊎ Fin n → Fin (m + n)
unsplit = [ inject+ n , raise {n} m ]

I'd be happy to call it join or pack or a whole host of nicer names than unsplit.

Two things @MatthewDaggitt :

probably there should an issue filed to remove that duplication?
what I meant was the inverse function to splitAt, defined inline in https://github.com/agda/agda-stdlib/blob/f9a53e8699259c8567d374a31c2c5636efaf50d0/src/Data/Fin/Properties.agda#L484-L502 in the "proof" that Fin (m + n) ≃ Fin m ⊎ Fin n (but which doesn't exist itself anywhere).  i.e.

unsplit :: {m n : ℕ} (i : ℕ) → Fin m ⊎ Fin n → Fin (m + n)
unsplit = [ inject+ n , raise {n} m ]

I'd be happy to call it join or pack or a whole host of nicer names than unsplit.

Is now https://github.com/agda/agda-stdlib/pull/1393.

Is now https://github.com/agda/agda-stdlib/pull/1393.

Ah sorry, I was reading too quick and thought you'd written the opposite function.
Filed https://github.com/agda/agda-stdlib/issues/1395 for 1.

Ah sorry, I was reading too quick and thought you'd written the opposite function.
Filed https://github.com/agda/agda-stdlib/issues/1395 for 1.

This has been done, and the uses in the library of the specialized code removed.


NoRestart specified on machine causes script to fail

Testing on my home PC, I have NoRestart specified (not sure where?) so the machine didn't reboot after installing Windows Sandbox, which made the start-process line fail:
`
Verifying that virtualization is enabled in BIOS...
Checking to see if Windows Sandbox is already installed...
Windows Sandbox is not installed, attempting to install it (may require reboot)...
WARNING: Restart is suppressed because NoRestart is specified.
Please reboot to finish installing Windows Sandbox, then re-run this script...
Checking for latest version of foldingathome...
Using FAH v7.5.
Downloading latest folding executable: C:\Users\spenc\fah_conf\folding_installer.exe
Saving to C:\Users\spenc\fah_conf\folding_installer.exe...
Creating init command...
Saved Folding@Home configuration file to C:\Users\spenc\fah_conf\fah_sandbox_conf.xml
Directory: C:\Users\spenc\fah_conf

Mode                LastWriteTime         Length Name

-a----        3/26/2020   8:53 AM              0 fah_sandbox_conf.xml
Creating init command...
Saved logon script to C:\Users\spenc\fah_conf\init.cmd, this will be run upon starting Sandbox.
-a----        3/26/2020   8:53 AM              0 init.cmd
Creating sandbox configuration file to C:\Users\spenc\fah_conf\fah_sandbox.wsb
-a----        3/26/2020   8:53 AM              0 fah_sandbox.wsb
Starting sandbox...
Start-Process : This command cannot be run due to the error: The system cannot find the file specified.
At G:\GIT\Windows-Sandbox-Utilities\Folding In Sandbox\install_folding_sandbox_on_host.ps1:108 char:1

Start-Process 'C:\WINDOWS\system32\WindowsSandbox.exe' -ArgumentList  ...

  + CategoryInfo          : InvalidOperation: (:) [Start-Process], InvalidOperationException
  + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.StartProcessCommand

`

Added a pull request to exit the script if reboot is required to avoid getting this type of errors

PR approved; thank you! Closing.


Make testEnv AutoCloseable

🆕🐥 First Timers Only
This issue is reserved for people who have never contributed to Hedera or any open source project in general.
We know that creating a pull request (PR) is a major barrier for new contributors.
The goal of this issue and all other issues labeled by 'Good First Issue' is to help you make your first contribution to Hedera.
👾 Description of the issue
A gRPC channel (ManagedChannelImpl) is being garbage collected without properly shutting down, causing the following error:
SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=13, target=<IP_ADDRESS>:50211} was garbage collected without being shut down! ~*~*~*
Make sure to call shutdown()/shutdownNow()
java.lang.RuntimeException: ManagedChannel allocation site

This occurs because the testing environment is not consistently being closed, leading to open gRPC channels that are not properly terminated. Failure to shut down the channels can result in resource leaks and instability in the tests.
Proposed Solution:

To prevent this issue, make the IntegrationTestEnv class implement the AutoCloseable interface, so it can be automatically closed when the test finishes. This ensures that all gRPC channels are properly shut down, avoiding the need for manual calls to shutdown() or shutdownNow().

Additionally, the initialization of the testing environment should be wrapped in a try-with-resources block. This will ensure that the environment is closed automatically after the test, preventing resource leaks and the improper garbage collection of gRPC channels.

📋 Step by step guide to do a contribution
If you have never contributed to an open source project at GitHub, the following step-by-step guide will introduce you to the workflow. More information and concrete samples for shell commands for each step can be found in our CONTRIBUTING.md file.
A more detailed general documentation of the GitHub PR workflow can be found here.

[ ] Claim this issue: Comment below that you are interested in working on the issue
[ ] Wait for assignment: A community member with the given rights will add you as an assignee of the issue
[ ] Fork the repository: You can do that in GitHub (by simply clicking the 'fork' button).
[ ] Check out the forked repository
[ ] Create a feature branch for the issue. We do not have a hard naming definition for branches but it is best practice to prefix the branch name with the issue id.
[ ] Solve the issue in your branch.
[ ] Commit your changes: Here, it is needed to add sign-off information to the commit to accept the "Developer Certificate of Origin" (https://developercertificate.org). More details can be found in our CONTRIBUTING.md
[ ] Start a Pull Request (PR): We have a pattern for naming pull requests that a GitHub Action checks. We use that pattern to support the creation of automatic release notes.
[ ] Check GitHub Actions: Several GitHub Actions will be triggered automatically for each PR. If a GitHub Action fails and you do not understand the cause of that error do not hesitate to add a comment to the PR and ask the Hedera developer community for support.
[ ] Wait for reviews: Members of the Hedera developer community will review your PR. If a reviewer finds any missing pieces or a problem, he or she will start a discussion with you and describe the next steps for solving the problem.
[ ] You did it 🎉: We will merge the fix in the develop branch. Thanks for being part of the Hedera community as an open-source contributor ❤

🎉 Contribute to Hacktoberfest
Solve this issue as part of the Hacktoberfest event and get a chance to receive cool goodies like a T-Shirt. 🎽
🤔 Additional Information
If you have any questions, just ask us directly in this issue by adding a comment. You can join our community chat at Discord. A general manual about open-source contributions can be found here.

hi there, i would like to contribute this!

just to get this straight: replacing the constructor call and associated call to the close()-method (416 occurrences in 217 files) with a try-with-resources block is part of this issue?

Oh, let's ask @0xivanov if we can split that up. Having 416 code places changed is a lot for a good first issue ;)

While touching multiple places, it is actually a good fit for beginners because the changes involve simple, well-defined adjustments rather than deep modifications in the SDK’s core functionality.

The solution primarily involves adding an AutoCloseable implementation and wrapping existing code in try-with-resources blocks. These are straightforward modifications that don't require deep knowledge of the gRPC internals or complex SDK implementation details.
The changes are isolated to test environments, meaning they're not touching the core logic of the SDK. It provides a clear learning opportunity without risking breaking critical parts of the system.
The task is mostly repetitive, which can be helpful for newcomers. Once they understand the pattern, they can apply it across the codebase, gaining confidence while having a clear direction.

Its not a big Deal for me, i will not do this manually 😀 will try out vscode search&replace functionality with some regex magic to get it done, probably this week i will get to work on this

Great! I will assign you tomorrow when I’m back at my laptop @steffenboe

@hendrikebbers assigned!

fixed in #2023


feat: local models using ollama and openai-compatible external models

tested locally with mistral using ollama.
needs further testing. External models compatible with OpenAI work fine.

related to #3
cc @granawkins