OV5642 image output is torn and blocky

Hello,
I am using the ArduCAM 5MP (Rev B) and I am having terrible trouble getting anything useful out of it. I can talk to it via I2C, and talk to it via SPI. I can start captures, read the FIFO length and clock out data. I can see it all on an oscilloscope and it looks great. When I reassemble the bytes, I just get garbage though. Anyone familiar with the issue, seen it before or anything?
I am not using an Arduino, but writing standard C on a PIC32. Everything else is working. The full app is reading chunks of data from the SPI bus and transferring via UART to a cellular transceiver which is FTP-ing the data to a server. I can transfer random data chunks with known data from the micro and receive the exact same (correct) bytes on the FTP server. So I am inclined to believe that the problem is somewhere in the ArduCAM. Either is setup or execution.
Any help is appreciated, this one part has blown out my schedule and I am on the brink of dropping it and getting a new module...

I see in another issue ticket with a rely from April 20th, 2016 where it was said:

@Juan
There is SPI timing error with 5MP when used in ESP8266, please try our special library released for ESP8266 from https://github.com/ArduCAM/ArduCAM_ESP8266_UNO/tree/master/libraries/ArduCAM

"SPI Timing Error" is interesting... Is the waveform from the datasheet incorrect? Is there more of a minimum hold time before clocking than lead to believe? I am running around 2Mhz, which I thought should be well below the 8MHz rate the datasheet says it is capable of.

Looks like the special timing thing in the ESP code is the "yield" arduino function to keep the background tasks going. That's obviously not my problem.
Is there any way for me to write a pattern to the FIFO and then read the pattern back to see if it is skipping or something???

@NVergunst
Hi,
Please try to narrow the data buffer?
Let us know if you need more help。
Regards,
ArduCAM support team.

Narrow the buffer on which end?
I have tried with reading 1K at a time, 100bytes at a time, and even 1 byte at a time. It seems like if I read 100-250 bytes at a time, I get the most image out. If I read 1 byte at a time, the image doesn't even become a jpeg.

Is there a way for me to read the FIFO pointer from the ArduCAM? Make sure it is being set to index + bytes read and/or set it back and re-read data to verify integrity?

@NVergunst
Hi,
Our camera support point reset.
You can refer to the below code.
uint8_t read_fifo_again(ArduCAM myCAM)
{
uint8_t temp, temp_last;
uint32_t length = 0;
temp = myCAM.read_reg(ARDUCHIP_FIFO);
myCAM.write_reg(ARDUCHIP_FIFO, temp|FIFO_RDPTR_RST_MASK);  //FIFO_RDPTR_RST_MASK
length = myCAM.read_fifo_length();
Serial.println(length, DEC);
if (length >= FIFO_SIZE) //8M
{
Serial.println("Over size.");
return 0;
}
if (length == 0 ) //0 kb
{
Serial.println("Size is 0.");
return 0;
}
myCAM.CS_LOW();
myCAM.set_fifo_burst();//Set fifo burst mode
SPI.transfer(0x00);//First byte is 0xC0 ,not 0xff
length--;
while ( length-- )
{
temp_last = temp;
temp =  SPI.transfer(0x00);
if (is_header == true)
{
Serial.write(temp);
}
if ((temp == 0xD8) & (temp_last == 0xFF))
{
is_header = true;
Serial.write(temp_last);
Serial.write(temp);
}
if ( (temp == 0xD9) && (temp_last == 0xFF) ) //If find the end ,break while,
break;
delayMicroseconds(12);
}
myCAM.CS_HIGH();
is_header = false;
}
Let us know if you need more help.
Regards,
ArduCAM support team.

Is there a minimum time between back-to-back burst reads required? I'm not understanding why the image data is time variant. If I go slowly, I get more data out. But at speed, the signals look excellent and read back properly. It is just seems to skip. I will try to implement the SPI read of the FIFO address. Thanks for the quick support so far.

@NVergunst
Hi,
Please send your code to us and we will test it for you.
What's the speed of your SPI speed?
Let us know if you need more help.
Regards,
ArduCAM support team.

SPI Speed is roughly 2MHz. But I have slowed it down to 400KHz without change.

My code is written for a Microchip PIC32MX440F512H on a custom board. I doubt it will do much good. It's actually based off of the code here on github but all the routines were updated to work with the PIC peripherals and time slicing.

I just got my first real pictures out of the thing. I kept the SPI speed at 2MHz, but inserted 100ms delays between each 100bytes read. This gave me a full image out. Really really slowly, but it worked. So the problem is most definitely timing related...

@NVergunst
Hi,
Maybe it is related to the platform of the PIC32.The SPI
should not be configured well or have conflicts with other functions.
We have not tested it on the PIC32 platform.
Regards,
ArduCAM support team.

@NVergunst
Hi,
Maybe it is related to the platform of the PIC32.The SPI
should not be configured well or have conflicts with other functions.
We have not tested it on the PIC32 platform.
Regards,
ArduCAM support team.

Nope, the SPI is working perfectly. I am running a single thread operation with only 2 interrupts (I2C and UART). The way the program is made, the I2C interrupt is silent when the SPI is getting data since the master is the PIC32 as well. The UART could get interrupted at any time (likelyhood low) but the SPI doesn't care, it would just stretch a pulse. I've had this same type of code running at 40MHz+, I know it is not the problem.

What's the proper way of polling the device to know that the capture is complete and ready for readback from the FIFO? If I wait multiple seconds between reads, by inserting a breakpoint, it seems to transfer and image every time now. But if I let it free-run, I get garbage.

And any idea about the scan-lines in the captured image?

I have been having similar issues for months now. I have noticed that even when I reset the FIFO read pointer and read the same image multiple times, the data is different (although the number of bytes I receive is exactly the same). I can never get a good image from the Arducam Mini.
Inserting delays between reads doesn't seem to help in my case. Has anyone learned what the issue is more? I am fairly certain it is a timing issue on the Arudcam's side.

Since I have no "official" answer, in my case reading slowly with delays between transactions, gives me a workable image most of the time. In my tests, it's about 1 corrupt image in 350. This only applies to VGA quality images though, where the output JPG is roughly 80KB in size.  A full proper image is only uploaded maybe 1 in 90 times.
I am inclined to believe it is something speed/buffer related since the beginning of the image always looks correct. Always. Then as the image progresses it gets worse and worse. With the JPG encoding, it looks like basically it is missing information and then the rest of the image gets filled in.
I know my SPI code is rock solid. I have been using these things for over a decade and they are in production quality hardware. I am not a hobbyist playing with an Arduino, I'm a full EE/CE that has been doing this professionally for some time now.
And I solved my line problem above, you have to enable the line filtering in the OV module itself over the I2C. That datasheet is huge, but it has most of the info. The built in libraries supplied for the Arducam are pretty silly and just don't work for me. Maybe something magically in Arduino land happens to make it work, but in real embedded C land, they don't work. Only thing I didn't find in the OV datasheet or in the Arducam software is for auto-focus, as they say contact an apps engineer to find out how.

Thanks for your fast response!
I am also fairly certain that my SPI code and my I2C code is correct. I am also not using an Arduino. I'm doing this in bare metal with a SmartFusion2.
I was finally able to get a clean image from the Arducam. I had to insert a 20ms delay between every single byte that I read and send over UART. Because of this, it literally takes more than an hour to get an image. This speed is extremely slow. I need to be saving images within seconds.
So there is definitely a timing problem with reading the image from the buffer. @supprot @ArduCAM How long does it take for the next byte to be in the read FIFO? I think this is the problem as the beginning of my images are always fine as well, but then they get messed up during the middle of the image and the jpeg ends up smaller than it should be.

What has been mostly working for me, is a 2MHz SPI, reading roughly 125bytes at a time, with a variable (software state machine) delay between burst reads between 5ms and 10ms.
I am guessing the way that the buffer code is written, when you request a pixel in a different page of memory, there is some non-zero delay required to allow the controller to catch up. In Arduino land where everything is dog slow, the user would never see these types of problems.

Thanks. My code is using a 6 MHz SPI and the pictures I'm getting are in total around 200 kB.
I might do some more testing and I'll post here if I learn anything useful.

Ok, the logic inside the ArduCAM is DEFINITELY messed up. It is not my code. Verified by a super awesome calibrated scope. My software, my independent analyzer, my scope screenshots, and my scope's built in serial decode all agree. The only outlier is the ArduCAM. That's pretty telling.
I have it in a loop with a 2.2MHz SPI reading back the ID register which according to the datasheet will always 100% return 0x62. Well, it doesn't.
These 3 screencaptures show the reading of 0x64. Somehow it has skipped a clock and is outputting the wrong value, a clock too early. First image shows the previous transaction (good) plus the wrong one. Then zooming in on the bad 0x64 read. You can faintly see the cursors. I lined those up manually at the end of the first nibble to the bad data middle clock. Somewhere in that section the ArduCAM internally registered an additional clock. As you can see from the much more zoomed in capture, there is no spurious clock, and it looks the same as all the other reads.

And here is an even worse one! A readback of 0x00! It didn't even try...

There is something really wrong with the SPI interface. When transferring huge data like an image, small problems like this in pixels is hardly noticeable. But when it happens often... corruption.

And it happens at 64KHz... Does not seem to be a speed problem. Read 0xC4 back which is 0x62 shifted a clock.

I've been getting good stable reads at 256KHz SPI Clock with 10ms between reads. Crazy slow obviously, but right now I want to get it robust, then ratchet it in to find where it breaks.

At 1.5MHz with 40us between transactions, I get about 1 error per 20K transactions. Not good for a large image...

@NVergunst
Hi,
please try improve sampling frequency.
Let us know if you need more help.
Regards,
ArduCAM support team.

Sampling us NOT the issue. This is a very deep memory scope and 2Gs/s. Believe me, that's not the issue.

@NVergunst
Hi,
That's a strange phenomenon.
We just do a test and read the OV5642 ID register  0x300a in the loop.
We set the SPI speed 4MHZ.
The data should be 0x56, if error , break.delay(3000) and it can work fine.
You can try the test code as below:
#include <Wire.h>
#include <ArduCAM.h>
#include <SPI.h>
#if !(defined ESP8266 )
#error Please select the ArduCAM ESP8266 UNO board in the Tools/Board
#endif
// set GPIO16 as the slave select :
const int CS = 16;
ArduCAM myCAM(OV5642, CS);
void setup() {
uint8_t vid, pid;
uint8_t temp;
Wire.begin();
Serial.begin(115200);
Serial.println(F("ArduCAM Start!"));
// set the CS as an output:
pinMode(CS, OUTPUT);
// initialize SPI:
SPI.begin();
SPI.setFrequency(4000000); //4MHz
}
void loop() {
while(1){
myCAM.rdSensorReg16_8(OV5642_CHIPID_HIGH, &vid);
Serial.print("The vid is");
Serial.println(vid,HEX);
if(vid !=0x56) break;
yield();
}
Serial.println("The SPI interface is error !!!");
delay(3000);
}
Let us know if you need more help.
Reards,
ArduCAM support team.

I don't have an Arduino hooked up to it, we use real hardware. Hardware that is proved to work in real embedded systems. You can see very plainly in the oscilloscope screenshots that a valid SPI request is going across the wire and it usually reads back 0x62. However sometimes it reads back 0x64 or 0xC4 or 0x00 or some other variation of bit shifting or just errors.
You can see the waveforms. Captured by a calibrated expensive real lab hardware Agilent MSO scope.

Well, back to getting garbage readings again. The data coming out of the SPI buffer is identical to my scope readings and what shows up in my micro's RAM. It's just not a valid image. I even installed a brand new from the box secondary unit. Same problems.

I don't know if this is similar, I set up two arduinos, one to work with the OV2640 mini, and the other to save to SD, and also carry out some other functions. I read the data from FIFO, then sent the byte data to the other arduino uno chip via Serial (SoftSerial). All was successful except that, the data received on the other arduino end which I saved to SDCard was though a valid JPEG file, was corrupted.
Also the nature of the byte data on both chip was different... the byte data was different no matter what I did.
So I just dumped the idea and found another way around to harvest the data.

@ayoadejoe I'm experiencing same/similar problem, did you managed to figure it out?

@NVergunst I'm experiencing same/similar problem, did you managed to figure it out?

@NVergunst I'm experiencing same/silimar problem, did you manage to figure it out?

@NVergunst I'm experiencing same/silimar problem, did you manage to figure it out?

I abandoned the ArduCAM and switched to uCAM-III.  I make custom hardware all the time, and they don't believe it is possible there is anything wrong with their hardware ever. Magically when I switch to a different camera everything works exactly as it should. Meaning everything on my end is and was correct. Blaming it on my SPI peripheral because it is not in Arduino-land is ridiculous and obviously not the problem as I can use the SPI port to read and write megabytes to filesystems.
The ArduCAM not only cost me a couple hundred bucks in hardware cost, but months in dev time. And it is 100% the ArduCAM problem. Drop it now, it will never work properly in any full system. It may be good enough for a noobie with a high school project, but it won't exist in real hardware.

@NVergunst thanks for your quick response, I'll check out uCAM-III. STM32F103 here and I'm using hardware SPI (used the same configuration previously for many peripherals, LCDs etc without problem at all). I tried changing SPI frequency, number of bytes to read, delays between those reads, different JPEG sizes - corrupted JPEG will be transfered sooner or later no matter what. Ehh...

That's why I captured the oscilloscope images. The SPI port (my peripheral like yours) is reading back exactly what is being put out on the bus. The problem is the data on the bus is wrong, that simple. The uCAM-III is a much lower resolution camera, but the interface has so far been solid. I can reliably get images out.

I'm sorry to hear that there is some problem using our cameras. I very regret to miss this thread from my email list. Our camera and examples are well tested on Arduino board, if you need to port to other platforms you have to contact us for help.
The following timing diagrams show the complete procedure how ArduCAM 5MP Plus (OV5642) module works:

Check the sensor model ID:
OV5642 I2C slave address  0x78 for write, 0x79 for read, OV5642 model ID register is 0x300A=0x56 and 0x300B=0x42

SPI operations based on Arduino UNO board at 4MHz SPI speed.
2.1 SPI Bus Test
Write 0x55 to 0x00 address and read the value from 0x00 address.

2.2 Image Data Transfer
After issue a capture command, pooling the capture completion flag by reading 0x41 register until euqals to 0x08.

Then read the image length from register 0x42~0x44, this is total length of the valid image data

Next issue burst FIFO read command 0x3C and followed by bytes read from the FIFO, in the following diagram the 0xFF, 0xD8 are read from the beginning which is JPEG header in this case. Note each burst read cycle should be started with 0x3C command, followed by any numbers of data to be read. Here read 256 bytes in this example. When reading data, you have to send a dummy data over MOSI line, here sending 0x00 for example.

JPEG tail can also be read from the end of the valid data, 0xFF, 0xD9 in this case.

2.3 Complete capture and read timing diagram
The CS pin only active low during the 256 bytes burst read, CS pull high when send the 256 bytes over the UART. One frame has multiple burst read cycle.

Very important code in the burst read function
/**important code in the burst read function/
myCAM.CS_LOW();
myCAM.set_fifo_burst();
while ( length-- )
{
temp_last = temp;
temp =  SPI.transfer(0x00);
//Read JPEG data from FIFO
if ( (temp == 0xD9) && (temp_last == 0xFF) ) //If find the end ,break while,
{
buf[i++] = temp;  //save the last  0XD9
//Write the remain bytes in the buffer
myCAM.CS_HIGH();
Serial.write(buf,i);
is_header = false;
i = 0;
}
if (is_header == true)
{
//Write image data to buffer if not full
if (i < 256)
buf[i++] = temp;
else
{
//Write 256 bytes image data to file
myCAM.CS_HIGH();
Serial.write(buf,256);
delay(5);
i = 0;
buf[i++] = temp;
myCAM.CS_LOW();
myCAM.set_fifo_burst();
}
}
else if ((temp == 0xD8) & (temp_last == 0xFF))
{
is_header = true;
Serial.println(F("ACK IMG"));
buf[i++] = temp_last;
buf[i++] = temp;
}
}

Arducam host application

Conclusion
We tried our best to implement read to use Arduino examples for user without a line of code.
If you have problem to modify our examples or port to other platforms, you need to email us for support. There may be some tricks during the development. Thanks for your understanding.

Yes I get all that, and did all that in the code. As you can see, I was able to pull images. Pulling an image once was not a problem as some times it would work. The problem was in reliability. I would get bad data back from the module on the SPI line. The only suggestions were that my SPI peripheral was corrupt or bad, my entire architecture choice was bad, etc. And when I provided a screenshot as proof of the camera sending back bad data, the only suggestion was that my scope was not sampling at a high enough frequency or something else. Never is it the ArduCAM fault.
I am using professional high end equipment, and one of my previous jobs was a Validation Engineer to break digital interfaces attached to power regulation silicon that go into high end systems. IC level stuff. I am not just some dude trying out Arduino for the first time. I am using real hardware, in a real system, in real conditions, and the digital interface fails.
Look at the May 20th post with the screenshot of the scope reading back 0xC4. That according to the datasheet is a read only register that ALWAYS returns 0x62. I should be able to read it at some maximum clock frequency back to back to back to back forever, and never get anything other than 0x62. Well there is plain view is a perfect 0xC4 read-back captured on analog waveform channels on a high end scope at 64KHz. I'd like an explanation of that first.

@NVergunst I understand that you are real engineer, I respect your feedback. I'm not saying there is no problem with ArduCAM hardware ourself. We used to have some hardware issue, like bit rotation issue on eariler version of 5MP cameras, and we have to use software work around to fix it. But now the new version hardware 5MP-Plus is much more stable.
There is some tricks on programmig due to the complex burst reading bus timing, it will easily cause problem when break an entire image read into small bunches blob data burst read. You can also try single read command to see if it is stable.
And we also find that the JPEG engine inside OV5642 sensor sometimes breakdown that cause broken JPEG data, maybe with the buffer overflow during compression.
Also if you send the data over UART, you have to add some delay between each serial transfer.
Once again if you use other MCU other than Arduino, which we are not fully tested, you can contact us to work together. If you still have chance to try our new version, we can send you free sample for further testing.

@NVergunst I understand that you are real engineer, I respect your feedback. I'm not saying there is no problem with ArduCAM hardware ourself. We used to have some hardware issue, like bit rotation issue on eariler version of 5MP cameras, and we have to use software work around to fix it. But now the new version hardware 5MP-Plus is much more stable.
There is some tricks on programmig due to the complex burst reading bus timing, it will easily cause problem when break an entire image read into small bunches blob data burst read. You can also try single read command to see if it is stable.

Unfortunately can't do a single read of a large image because I don't have the RAM, that's the reason I went with an ArduCAM and now a uCAM in the first place. I need something with the big RAM for images so my processor can fetch it slowly and pass it out over a cellular connection. When I got the image down to VGA quality I can store it in RAM but I was still getting tearing/blocking even with single read commands.

And we also find that the JPEG engine inside OV5642 sensor sometimes breakdown that cause broken JPEG data, maybe with the buffer overflow during compression.

Well that would be good to know and maybe detect. Since you are wrapping the OV5642 digital firmware with your own, it would be good to know if a JPEG capture has the correct header and is at least the correct size or something like that. I obviously have no way of knowing if the data returned by the ArduCAM was corrupt because the data returned to it by the OV5642 was corrupt. However, this happened almost all the time, it tells me that's unlikely. Also the fact that reading an ArduCAM register that has no equivalent in the OV5642 was sometimes corrupted makes me believe the ArduCAM FPGA was causing corruption. Did you see the scope shot of the bad data from a read only register that always returns 0x62 not returning 0x62 but bit-shifting it?

Also if you send the data over UART, you have to add some delay between each serial transfer.

Everything was done over SPI. I also saw no mention of minimum time between packets in the datasheet for the SPI interface.

Once again if you use other MCU other than Arduino, which we are not fully tested, you can contact us to work together. If you still have chance to try our new version, we can send you free sample for further testing.

There should be no reason for something with a universal interface like SPI to only work if you use a specific compiler. If so, then the compiler is doing is some magic in the background that should be in your datasheet because it is no longer generic SPI.

@ArduCAM to sum up - you have virtually no datasheet. Scopes which you provided should be in much more detail provided in DS from day one. Like said in Halt and Catch Fire - "Software goes, hardware stays".

@ArduCAM to sum up - you have virtually no datasheet. Scopes which you provided should be in much more detail provided in DS from day one. Like said in Halt and Catch Fire - "Software comes and goes. Hardware is forever.".

@pawcuq
This is the datasheet for Arducam www.arducam.com/downloads/shields/ArduCAM_Mini_5MP_Plus_OV5642_Camera_Module_DS.pdf
http://www.arducam.com/downloads/shields/ArduCAM_Software_Application_Note.pdf
www.arducam.com/downloads/shields/ArduCAM_Mini_5MP_Plus_Camera_Shield_Hardware_Application_Note.pdf

@NVergunst
I guess you got the old 5MP camera, can you send me the photo to confirm.
We can send you a free new camera for your double checking, or you can send us your testing code, we can help you test here.
Thanks.

Hi.
I got a 5MP Mini OV5642 module that I purchased recently. I see the software in an Arduino UNO works with the supplied Host App. But when I try to make my own app  I get the same behaviour as described above. I wonder what the problem could be. And I wonder if I can take a look at the source code of the Host App. I see, it really changes a lot of settings in the serial port before it starts transmitting.


feat/101-Contributors-Path-DataGrid

What does this PR do?
Changed table of contributor path to datagrid component.
Some imports are missing because we have the contributor path incomplete.

How should this be manually tested?
Select one project with member and look for the Contributors section
Screenshots

Checklist

[ ] I added the necessary documentation, if appropriate.
[ ] I added tests to prove that my fix is effective or my feature works.
[ ] I reviewed existing Pull Requests before submitting mine.

I like it that it has dark mode, but it doesn't show the values as nice as the table, I don't like it that I can't see all the skills or roles of the team members. Also, the names were a link with their emails, which allowed printing both email and name on the same column 😛 . Do you think those shortcomings can be fixed with the DataGrid?

Dark mode looks good. But I'm not able to see the Intro, Setup, Quick Win and Major Contributor values. Am I missing something?:

@maubarraganwize We still need to migrate that part of the previous project 😅

Can we make the datagrid sort by clicking the name column? Also, can we make the table have the full height instead of presenting a vertical scrollbar?
I'm still not 100% convinced this datagrid is better than our HTML table approach. For example, I like that the HTML is very easy to copy paste for example, into a spreadsheet. Seems like this is not possible with the datagrid.
Now that you've implemented it. Do you think it is an improvement over the regular HTML table?


Problem launching the CloudFormation stack

Hello!
I was trying to follow the implementation guide and two errors came up and I don't know how to continue. Attached capture of the error at the time of clicking on the button "Launch Stack":

Any idea what may be the mistake I am making?
Thanks in advance
Sebastian Torres

Hi Sebastian - I confirmed the S3 Bucket containing the CloudFormation template is publicly accessible. Please try launching the stack again and let me know if you encounter any challenges.


new structure added without announcement?!

what is this? "Mojang" XD

I think I know what this is, it's a scam. Build it yourself didn't you?


Pressing "delete" button in explorer leads to "Pattern matching" but not to actual file deletion.

VSCode Version: 1.36.0
OS Version: Ubuntu 19.04

Steps to Reproduce:

Select any file in explorer
Press "delete" button

Workaround:
rmb -> delete still works, but very inconvenient.

Does this issue occur when all extensions are disabled?: Yes

This is what I've meant.

/duplicate of #76514


Add KnownBits support for souper2llvm

For this souper:
%0:i64 = var (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx000000)
%1:i64 = addnw 63:i64, %0
%2:i64 = lshr %1, 6:i64

I think this is a reasonable translation:
@glob = external global i64, align 8
define i64 @foo1(i64 %x) {
  %1 = and i64 %x,<PHONE_NUMBER>3709552000
  store i64 %1, i64* @glob
  %2 = add nsw nuw i64 63, %1
  %3 = lshr i64 %2, 6
  ret i64 %3
}

The purpose of the store is to ensure that the masked value has another user, which should help keep LLVM from messing with the masking code, as opposed to the code following the mask.

Looks good. I'll hack it on Friday.

Not sure if there's anything here for you to steal but here's my crappy program for computing masks.
#!/usr/bin/python

import fileinput
import os

for line in fileinput.input():
    line = line.rstrip(os.linesep)
    rline = line[::-1]
    zeromask = 0
    onemask = 0
    all = 0
    bit = 0
    for c in rline:
        all += 1<<bit
        if c == '0':
            zeromask += 1<<bit
        if c == '1':
            onemask += 1<<bit
        bit += 1
    print "for " + str(bit) + " bit value " + line
    if zeromask != 0:
        zeromask = zeromask ^ all
        print "val = val & " + str(zeromask)
    else:
        print "no zero mask needed"
    if onemask != 0:
        print "val = val | " + str(onemask)
    else:
        print "no one mask needed"

Thanks. You should start another competition in your blog post to do this in one line.


IPFS deployment speed

When you first deploy a shop to *.ogn.app it can take a minute or two to update due to having to provision an SSL cert as well as wait for the DNS to update. Also, if you try to visit the URL before the DNS is updated, you browser might cache the fact the DNS record doesn't exist causing another delay.
Perhaps we can cut down this time by:

Setting up a wildcard cert for *.ogn.app
CNAME *.ogn.app to IPFS prod server

I think it's also worth looking at ways to speed up the process of updating a site. Ideally this would be instant. Factors affecting it right now:

TXT record TTL time
Browser cache
Other stuff?

What techniques can we use to take the update time down to zero? Right now it's a fairly poor user experience: you click update, it tells you it's updated, but when you visit the site... it's not updated for a few mins, and only then if you clear your browser cache.

Also, if you try to visit the URL before the DNS is updated, you browser might cache the fact the DNS record doesn't exist causing another delay.

Since you're specifically mentioning ogn.app, what are we trying to improve here, exactly?  I don't think (?) most users are going to want their site running on our domain so I'm not sure what a wildcard cert is going to solve here.  So is it just users testing?  Could we maybe just send them to an IPFS gateway URL, hash and all(good bit of signaling here that it's on IPFS)?
DNS propagation delays aren't really a new thing.  It's gotten somewhat better than the past just because overall latency has allowed TTLs to be lowered(and TTLs are worth a review).  But my point is maybe we can take cues from other software that rely on it.

you click update, it tells you it's updated, but when you visit the site

I def think some low hanging UX update here is to tell them what's happening and how long it might take.  Properly informing users that DNS updates take time could also improve UX rather than expecting something to be instant.  We give them all the time to preview their site, we can show them a gateway URL, what's a few minutes for a production change over?

it can take a minute or two to update due to having to provision an SSL

We can use the priming technique here just like we're doing with IPFS gateways, but I don't expect a big positive difference.  The DNS changs must propogate to Let's Encrypt's DNS servers(and not be cached).  If the issuer makes the ACME request  early and LE(or their upstream) isn't updated, it could end up caching the old data for up to almost double the TTL(TTL of the new request landing the end of the TTL of the original old data).  Though this technique may still be worth trying so at least the user doesn't have to be the one to do trigger it.
Noticed you've also even using IPNS on some(?) of the sites.  In that case, IPFS updates don't even require DNS changes, just IPNS propagation.  How's that been working?

Browser cache

I suspect this is giving us more trouble than we expect.  When I was looking through logs I've been seeing lots of 404s on bundles and CSS.  I think it could be due to cached HTML pages pointing at bundles that no longer exist.  I'm not sure if we should include previous bundles in the newest bundle, or if there's some way to detect this update and reload(I've seen this done before on a react app but not sure how to do it myself offhand).

Most users will start off with a *.ogn.app domain before pointing their custom domain at it, so at least it helps solve the first time experience of deploying a shop. Also cuts down on the number of DNS records to manage if we have a wildcard pointing at prod IPFS.
The IPNS I've been using is DNS based... ie /ipns/store.ogn.app will cause IPFS to lookup the TXT record on store.ogn.app to get the hash... unless you're thinking of something else?
Agree on improving the UX to set expectations. But if there's a way we don't need to do that because the site updates immediately, that'd be better. This isn't an issue for our competitors so if there's a way we can get around the update time that'd be ideal. We could show the user the raw IPFS hash initially but that might freak out non-technical users.

Most users will start off with a *.ogn.app domain before pointing their custom domain at it, so at least it helps solve the first time experience of deploying a shop. Also cuts down on the number of DNS records to manage if we have a wildcard pointing at prod IPFS.

I can look into setting up a wildcard for that domain when I get a chance.  Might require yet another ingress but one way or another it should be doable.

The IPNS I've been using is DNS based... ie /ipns/store.ogn.app will cause IPFS to lookup the TXT record on store.ogn.app to get the hash... unless you're thinking of something else?

So, a record like this:
_dnslink.store.example.com. 300	IN	TXT	"dnslink=/ipns/example.ogn.app"

...makes IPFS look up TXT on _dnslink.example.ogn.app?  I thought IPNS pointed at a hash, not another DNSLink lookup?

But if there's a way we don't need to do that because the site updates immediately, that'd be better. This isn't an issue for our competitors so if there's a way we can get around the update time that'd be ideal. We could show the user the raw IPFS hash initially but that might freak out non-technical users.

Why wouldn't this be an issue for our competitors?  Setting up DNS for their site would have similar challenges.  At least on first use.
Maybe we need to break this whole GH issue down into individual component issues(DNS propagation, DNSLink+caching delays, issuer delay, ?).  Some might have solutions, some might need expectations management, some might require making serious compromises on the technology we use.

I thought IPNS pointed at a hash, not another DNSLink lookup?

It works recursively, so store.example.com -> /ipns/example.ogn.app -> /ipfs/Qmhash. Pretty neat.

Setting up DNS for their site would have similar challenges

Yea as you said first time can take a minute, but subsequent updates happen immediately which is hopefully what we can get to.
Yep makes sense to break this issue down, perhaps we can keep this one for discussion and link the new issues.

^ Created a bunch of issues ^
I did not create any regarding UX but think there should be at least one on informing users of what's going on during deployment and managing expectations.  I don't think we're even telling users how they might configure their own domain right now, so I think this is probably something we might want to add.  This might also link up with #118 and we can tell them "We're configuring your store now please wait..." and when it's done and working then we can link them to their store instead of linking them to an incomplete deployment like we're doing now.

This could be useful when trying to figure out when deployment is complete:
$ curl -v -X POST http://localhost:5001/api/v0/dns?arg=store.example.com
*   Trying ::1:5001...
* connect to ::1 port 5001 failed: Connection refused
*   Trying <IP_ADDRESS>:5001...
* Connected to localhost (<IP_ADDRESS>) port 5001 (#0)
> POST /api/v0/dns?arg=store.example.com HTTP/1.1
> Host: localhost:5001
> User-Agent: curl/7.70.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
< Access-Control-Expose-Headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
< Content-Type: application/json
< Server: go-ipfs/0.5.0
< Trailer: X-Stream-Error
< Vary: Origin
< Date: Tue, 23 Jun 2020 17:55:00 GMT
< Transfer-Encoding: chunked
<
{"Path":"/ipfs/QmRza91bPe6dxzu9dkYGoq27DtTPKhXza3Q2VBju47eCdB"}


Add option to set custom client properties on connection

I mainly use this to identify which service a specific connection belongs to.

@luddd3 - would you mind maybe adding an example of your use case to the README just so others can see how to leverage this?

I have tried to describe it as well as I can in the README now =)


Write Swagger tests for Questionnaire endpoints

Write Swag test for Questionnaire endpoints to check working of API for all the endpoints present in Questionnaires Controller

Added tests to test all CRUD operations defined in Questionnaire controller


Stage issues for users to work through into gitlab during setup

I'm working on this now as part of NetDevOps Live Preperation.


Pass sysroot to boost compiler for internal link tests (cross compila…

…tion scenario)

This change will break Visual Studio build. See: https://github.com/ruslo/hunter/commit/af66eeb51988e4e09b6d1895a0f8a314703ecf3e


remove last updated backported to 2.5

Description
Removed "last updated by" sections from the UI as the SearchMonitor API is no longer intended to return that info. See the issue below for more details.
I have backported for 2.5
Issues Resolved
#763
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Manual backport of PR https://github.com/opensearch-project/alerting-dashboards-plugin/pull/767

Hi @AWSHurneyt
The E2E tests are failing. I think they are failing because cluster_metrics_monitor_spec.js is missing an import sample_cluster_metrics_monitor and monitors_dashboard_spec.js is accessing an element that does not exist in the DOM, probably due to some delay in loading the application. I think this could be a flaky issue.
Please guide me on the next steps.
Thanks,
KC


How does storageKey/LocalStorage (getItem and setItem) work in MUI Datatable?

Hi,
If someone can help me understand How does storageKey/LocalStorage (getItem and setItem) work in MUI Datatable that would be really great?
What I am trying to achieve is lets say I have a column with name:"title", display:"false".
Now by default "title" will be unchecked in viewColumns. When I check that checkbox, "title" should get added in localstorage apart from getting added in the table column. When I log back in, then I should still see "Title" added in the table . It should not disappear just because I logged out of my session. How do I localstorage.getItem here? What does storageKey do?
P.S - I am able to persist that column for that session only using onViewColumnChange function.

I was able to figure this out myself. Here is the code . Hoping it will help someone else.
import React, { useState, useEffect } from 'react';
import MUIDataTable from 'mui-datatables';
function App() {
const [tableState, setTableState] = useState({});
useEffect(() => {
	const storedData = localStorage.getItem('my-table-state');

	if (storedData) {
		setTableState(JSON.parse(storedData));
	}
}, []);

useEffect(() => {

	localStorage.setItem('my-table-state', JSON.stringify(tableState));
}, [tableState]);

const columns = [
	{
		name: 'Name',
		options: {
			filter: false,

			display: useEffect(() => { }, [])

		}
	},
	{
		name: 'Title',
		options: {
			filter: false,
			// display:true

		}
	},
	{ name: 'Location', options: { filter: true } },
	{ name: 'Age', options: { filter: true } },
	{ name: 'Salary', options: { filter: true } },
];

const data = [
	['Gabby George', 'Business Analyst', 'Minneapolis', 30, 100000],
	['Aiden Lloyd', 'Business Consultant', 'Dallas', 35, 200000],

];

return (
	<MUIDataTable
		title="Employee List"
		data={data}
		columns={columns}
		options={{
			filterType: 'checkbox',
			storageKey: 'my-table-state',
			draggableColumns: {
				enabled: true
			},
		}}
		state={tableState}
		setState={setTableState}
	/>
);

}
export default App;

Thanks for the help it worked for me too, I just have a question I used your solution to persist the filtered data I want, but when I change the page and come back the data persisted but the filter did not. Did you understand? can you help?


Last CardItem text cut off in DeckSwiper component

Issue Description
Placing a Deck Swiper inside a react-native <View> with flex:1 results in last CardItem text being cut off on
certain screen sizes.
node, npm, react-native, react and native-base version, expo version if used
    "native-base": "2.0.12",
    "react": "~15.4.2",
    "react-dom": "~15.4.2",
    "react-native": "0.42.0",
    "react-native-vector-icons": "^4.0.0",

Expected behaviour
Bottom card item would not be cut off as in screenshot. Or as a workaround that a ScrollVIew could be wrapped around the CardItem.
Actual behaviour
Last CardItem text in the DeckSwiper is cut off inside View instead of resizing for given device screen size.
Steps to reproduce
Review the following gist of the component -
https://gist.github.com/BrianJVarley/8f6a8ea5ab0fa2ca699f9546165808f2
Is the bug present in both iOS and Android or in any one of them?
Android
Any other additional info which would help us debug the issue quicker.
A similar issue was opened in regards to wrapping the component in a ScrollVIew but no solution given:
https://github.com/GeekyAnts/NativeBase/issues/867
Example of CardItem cut off in view

I am experiencing this issue too, where DeckSwiper cut off some text and i can't scroll up to show the cut off text. Have you gotten a fix for it?

Hi @BrianJVarley , you can use <ScrollView> from react-native for scrolling.
<DeckSwiper
            dataSource={cards}
            renderItem={item =>
              <Card style={{ elevation: 3 }}>
                <CardItem>
                  <Left>
                    <Thumbnail source={item.image} style={{backgroundColor: 'red'}}/>
                    <Body>
                      <Text>{item.text}</Text>
                      <Text note>NativeBase</Text>
                    </Body>
                  </Left>
                </CardItem>
                <CardItem cardBody>
                  <Image style={{ height: 300, flex: 1, backgroundColor: 'red' }} source={{uir: item.image}} />
                </CardItem>
                <CardItem>
                  <Icon name="heart" style={{ color: '#ED4A6A' }} />
                  <ScrollView style={{height: 75}}>
                    <Text>{item.name} Contrary to popular belief, Lorem Ipsum is not simply random text.  Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source.</Text>
                  </ScrollView>
                </CardItem>
              </Card>
            }
/>

Hi @BrianJVarley, closing this now due to inactivity. Let us know if the issue still persists.


specify the encoding is UTF8

JSON and YAML suport other encodings (at least utf-16 and utf-32). I'm not sure if we can be agnostic here, or not? In any event, Python on Windows doesn't default to a sane encoding, so we can expect unicode-related annoyances there.

I think this is the way to go.
What does Python do on Windows, btw?

I'm thoroughly confused about Windows, and localisation in general. But see
https://docs.python.org/3/using/windows.html#utf-8-mode
and
https://vstinner.github.io/python37-new-utf8-mode.html

I would just stay away from this minefield - I don't see what problem we're fixing by having an opinion about it. It's up to implementations at the end of the day what text encodings they support.

The problem (I think) is that one cannot, in general, determine the encoding of a given file automatically. So we must say "let's interpret this file as utf-8" (or some other encoding) when it is opened/read. Some non-ascii bytes are valid in multiple encodings, but have different interpretations in those distinct encodings. On MacOS and Linux, files are read as utf-8 by default. On Windows, it seems that by default this is dictated by the current locale setting.
I think this means that someone could write a yaml file on Linux or MacOS using a unicode name, and the file would then be decoded on Windows using a mystery encoding where the non-ascii bytes are potentially not even valid (and thus the file would be unreadable).
On the other hand, if we say all yaml and json files must be utf-8, then any parser that exhibits this problem on Windows is simply non-conformant. Not only that, there is a simple fix (explicitly open the file as utf-8).

I don't think we can do much about the text encoding that people use, and even if we did go to the trouble of specifying an encoding it wouldn't do any good.  Let's not worry about it.

Sorry, I think I misunderstood here - if it's just specify an encoding in our code so that the tests pass, then totally, let's do that.
I thought you meant, "let's talk about text encodings in the specification and require that people use utf8".

Hmm, well we have to do the latter if we do the former. In demes-python we want to test against the spec test cases, which include files with unicode in deme names. To make the tests pass (https://github.com/popsim-consortium/demes-python/pull/430) we thus need to open the files with encoding="utf8" --- not just in the tests themselves, but in the code path that users use to load files into demes-python. Subsequently, this means that users will have to write their yaml files with utf8 as their encoding, because their yaml files won't be loaded correctly otherwise.
Yeah, its a pain. The alternative, I guess, is to leave this all alone (skip tests on Windows), and then we have Windows users writing yaml files with mystery encodings that no one else can read (possibly not even Windows users with other locales) without reverse engineering the encoding or renaming the deme names.

I mean we can write it into the spec if we like, I just don't think it'll make any difference to whether people encounter the problems in practise. Text encoding is a problem beyond our control.

I think you're right @jeromekelleher. Let's open files as utf-8, but avoid saying anything about the encoding in the spec. Whatever we do, something will break for someone, so lets wait for someone to complain.


VersionedAggregate and polymorphic classes problem

When using VersionedAggregate (handling an aggregate of type e.g. BaseClass) it seems not to properly create an instance of ClassA which disinherited from BaseClass correctly so that validateUpdate on VersionedAggregate fails with a VersionedAggregate$AggregateDoesNotExistException.
When using the attached Scala class these commands fail:
val polymorphicActor = system.actorOf(PolymorphicActor.props("4711", endpoint.id, eventLog))
polymorphicActor ! CreateClassB("4711", "Test")
polymorphicActor ! SetPropertyTwo("4711", 2)
PolymorphicActor.scala.txt

In your PolymorphicActor#onEvent when handling ClassXCreated-events you do not modify the actor's internal state (polymorphicAggregate). It seems when you overwrite your internal state like follows:
    case e: ClassACreated =>
        polymorphicAggregate = polymorphicAggregate.handleCreated(e, lastVectorTimestamp, lastSequenceNr)

That SetPropertyTwo is handled successfully.

Yes, you are right! For whatever reason I have overseen that.
Many thanks for your quick answer!

You are welcome.


Email link to be fixed

5.Email Issue - When emails are sent to mentee's or mentor's after booking, they receive a link in the email and link is incorrect: "mentee-hi4-herokuapp.com" is the link they receive but it is not working, they must receive link "https://mentee-global.vercel.app/" (edited)

Updated hyperlinks in templates in SendGrid


Added missing previous selection discard upon the new selection change

Issue
Initially reported - https://github.com/recogito/text-annotator-js/pull/151#discussion_r1795661665

We need to restore the selection.clear() call if there's nothing to update. Because now the previous, "discarded selection" unexpectedly stays up until you create a brand-new one.
https://github.com/user-attachments/assets/cb0d6edc-5a48-400b-90f8-a61c41251200
*in my testing env, the annotations that don't have either a highlight or a note assigned get automatically garbage collected when they get unselected

Can be reproduced in the playground:
https://github.com/user-attachments/assets/81bf9fc7-380d-4438-a81e-d0d6365b251d

Thanks - yes, that makes sense. The previous selection should clear as soon as you start the next, not after you complete it.

I FOUND A REGRESSION IN THIS CHANGE
It breaks the existing annotations selection on Android! For an unknown reason, the order of the events is different on Android compared to the Mac or iOS when a user clicks over the text!

Mac
Android

So on Android when the annotation gets "clicked" in the pointerup handler - the selection gets immediately cleared in the selectionchange handler 😭. Which makes the selection of the existing annotations impossible.

Another mobile quirk!
On iOS the ordering is the same as on Android - the selection comes after the pointer events. BUTT! It fluctuates depending on where and how fast you click!
When you click over an annotation quickly - only pointerdown/pointerup are dispatched. But when slowly - all 4. When I tap over the textual content - all 4.
https://github.com/user-attachments/assets/cac92b06-5989-46d4-a7c8-9798723ea18f


Retry an entire suite

🚀 Feature Proposal
Retry a suite (file).
Motivation
I want to retry tests, even if there are hooks or tests with side effects.
Suites are the right level to retry in these scenarios because they are run independently from each other, and code within them is run in-order.
Example
jest.setRetryLevel('suite')

Pitch
Why does this feature belong in the Jest core platform?
Because retries are part of Jest. And this is not easily implemented outside Jest.

This would also be wonderful for my team. We have some suites with logic in beforeAll that should be ran before retrying a failing test.


Support PHP

🤔 What's the problem you're trying to solve?
Polyglot release should support all languages in the largest polyglot repo. Currently this is cucumber-expressions but will soon be Gherkin.
This includes PHP

Looks like PHP only uses git tags.
https://github.com/cucumber/common/blob/main/gherkin/php/default.mk#L29

I don't think there's anything to do here.

Then we should add it to the supported list.

Do you mean merging in #40 so it does an explicit noop? I wasn't sure why we needed that - will it error otherwise?

No it will not break, but it avoids confusion in the future. Because now we explicitly say that PHP should be a noop.

OK, so let's un-draft #40 and merge it?


Run with Local Docker Container RabbitMQ Service

Found in https://github.com/WIPACrepo/MQClient-RabbitMQ/pull/5
Running with a Docker container had problems connecting on my machine: pika.exceptions.IncompatibleProtocolError: StreamLostError: ('Transport indicated EOF',)
This includes the docker container running in run-with-wipactel-local.sh (and the commented-out container).
Priority unknown, this may not be worth the trouble if we plan on having a remote service. It could also just be my machine.

running with docker run -d -p 8084:5672 deadtrickster/rabbitmq_prometheus (docker run -d --rm -it -p 15672:15672 -p 5672:5672 rabbitmq:3-management was similar) produced:
2021-11-09 17:02:17 DellXPS-8BX90Z2 root[569781] DEBUG [__init__()] Initializing pub. (('localhost', 'queue1'); {})
2021-11-09 17:02:17 DellXPS-8BX90Z2 root[569781] DEBUG [connect()] Pub connecting...
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.utils.io_services_utils[569781] ERROR Socket EOF; <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('<IP_ADDRESS>', 48028)>
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.base_connection[569781] ERROR Transport indicated EOF.
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.base_connection[569781] ERROR connection_lost: StreamLostError: ('Transport indicated EOF',)
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.connection[569781] ERROR Probably incompatible Protocol Versions
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[569781] ERROR AMQPConnector - reporting failure: AMQPConnectorAMQPHandshakeError: IncompatibleProtocolError: The protocol returned by the server is not supported: ("StreamLostError: ('Transport indicated EOF',)",)
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[569781] ERROR AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: IncompatibleProtocolError: The protocol returned by the server is not supported: ("StreamLostError: ('Transport indicated EOF',)",); first exception - None.
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[569781] ERROR AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: IncompatibleProtocolError: The protocol returned by the server is not supported: ("StreamLostError: ('Transport indicated EOF',)",); first exception - None
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.blocking_connection[569781] ERROR Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: IncompatibleProtocolError: The protocol returned by the server is not supported: ("StreamLostError: ('Transport indicated EOF',)",); first exception - None
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.blocking_connection[569781] ERROR Error in _create_connection().
Traceback (most recent call last):
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
   raise self._reap_last_connection_workflow_error(error)
pika.exceptions.IncompatibleProtocolError: StreamLostError: ('Transport indicated EOF',)
2021-11-09 17:02:17 DellXPS-8BX90Z2 root[569780] DEBUG Creating new MessageGeneratorContext instance.
2021-11-09 17:02:17 DellXPS-8BX90Z2 root[569780] DEBUG [__init__()] Initializing sub. (('localhost', 'queue1'); {})
2021-11-09 17:02:17 DellXPS-8BX90Z2 root[569780] DEBUG [connect()] Sub connecting...
Traceback (most recent call last):
 File "examples/server.py", line 47, in <module>
   server(workq, resultq)
 File "examples/server.py", line 14, in server
   work_queue.send({"id": i, "cmd": f'echo "{i}"'})
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/wipac_telemetry/tracing_tools/spans.py", line 228, in wrapper
   return func(*args, **kwargs)
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient/queue.py", line 129, in send
   self.raw_pub_queue.send_message(msg)
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient/queue.py", line 74, in raw_pub_queue
   self._pub_queue = self._backend.create_pub_queue(
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 361, in create_pub_queue
   q.connect()
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 85, in connect
   super().connect()
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 45, in connect
   pika.BlockingConnection(
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
   self._impl = self._create_connection(parameters, _impl_class)
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
   raise self._reap_last_connection_workflow_error(error)
pika.exceptions.IncompatibleProtocolError: StreamLostError: ('Transport indicated EOF',)
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.utils.io_services_utils[569780] ERROR Socket EOF; <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('<IP_ADDRESS>', 48034)>
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.base_connection[569780] ERROR Transport indicated EOF.
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.base_connection[569780] ERROR connection_lost: StreamLostError: ('Transport indicated EOF',)
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.connection[569780] ERROR Probably incompatible Protocol Versions
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[569780] ERROR AMQPConnector - reporting failure: AMQPConnectorAMQPHandshakeError: IncompatibleProtocolError: The protocol returned by the server is not supported: ("StreamLostError: ('Transport indicated EOF',)",)
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[569780] ERROR AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: IncompatibleProtocolError: The protocol returned by the server is not supported: ("StreamLostError: ('Transport indicated EOF',)",); first exception - None.
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[569780] ERROR AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: IncompatibleProtocolError: The protocol returned by the server is not supported: ("StreamLostError: ('Transport indicated EOF',)",); first exception - None
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.blocking_connection[569780] ERROR Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: IncompatibleProtocolError: The protocol returned by the server is not supported: ("StreamLostError: ('Transport indicated EOF',)",); first exception - None
2021-11-09 17:02:17 DellXPS-8BX90Z2 pika.adapters.blocking_connection[569780] ERROR Error in _create_connection().
Traceback (most recent call last):
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
   raise self._reap_last_connection_workflow_error(error)
pika.exceptions.IncompatibleProtocolError: StreamLostError: ('Transport indicated EOF',)
Traceback (most recent call last):
 File "examples/worker.py", line 36, in <module>
   worker(inq, outq)
 File "examples/worker.py", line 14, in worker
   with recv_queue.recv() as stream:
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient/queue.py", line 205, in recv
   return MessageGeneratorContext(self._create_sub_queue(), self)
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient/queue.py", line 95, in _create_sub_queue
   return self._backend.create_sub_queue(
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 381, in create_sub_queue
   q.connect()
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 143, in connect
   super().connect()
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 45, in connect
   pika.BlockingConnection(
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
   self._impl = self._create_connection(parameters, _impl_class)
 File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
   raise self._reap_last_connection_workflow_error(error)
pika.exceptions.IncompatibleProtocolError: StreamLostError: ('Transport indicated EOF',)

running with docker run -d --network=host deadtrickster/rabbitmq_prometheus without local wipactel produced:
2021-11-09 17:26:45 DellXPS-8BX90Z2 root[577491] DEBUG Creating new MessageGeneratorContext instance.
2021-11-09 17:26:45 DellXPS-8BX90Z2 root[577491] DEBUG [__init__()] Initializing sub. (('localhost', 'queue1'); {})
2021-11-09 17:26:45 DellXPS-8BX90Z2 root[577491] DEBUG [connect()] Sub connecting...
2021-11-09 17:26:45 DellXPS-8BX90Z2 root[577491] INFO Connecting with address=amqp://localhost
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.io_services_utils[577491] ERROR Socket failed to connect: <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('<IP_ADDRESS>', 50724)>; error=111 (Connection refused)
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[577491] ERROR TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP_ADDRESS>', 5672))
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[577491] ERROR AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[577491] ERROR AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused'); first exception - None.
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[577491] ERROR AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused'); first exception - None
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.blocking_connection[577491] ERROR Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused'); first exception - None
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.blocking_connection[577491] ERROR Error in _create_connection().
Traceback (most recent call last):
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
Traceback (most recent call last):
  File "examples/worker.py", line 36, in <module>
    worker(inq, outq)
  File "examples/worker.py", line 14, in worker
    with recv_queue.recv() as stream:
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient/queue.py", line 205, in recv
    return MessageGeneratorContext(self._create_sub_queue(), self)
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient/queue.py", line 95, in _create_sub_queue
    return self._backend.create_sub_queue(
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 377, in create_sub_queue
2021-11-09 17:26:45 DellXPS-8BX90Z2 root[577492] DEBUG [__init__()] Initializing pub. (('localhost', 'queue1'); {})
    q.connect()
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 139, in connect
2021-11-09 17:26:45 DellXPS-8BX90Z2 root[577492] DEBUG [connect()] Pub connecting...
    super().connect()
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 45, in connect
2021-11-09 17:26:45 DellXPS-8BX90Z2 root[577492] INFO Connecting with address=amqp://localhost
    self.connection = pika.BlockingConnection(
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.io_services_utils[577492] ERROR Socket failed to connect: <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('<IP_ADDRESS>', 50728)>; error=111 (Connection refused)
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[577492] ERROR TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP_ADDRESS>', 5672))
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[577492] ERROR AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[577492] ERROR AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused'); first exception - None.
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.utils.connection_workflow[577492] ERROR AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused'); first exception - None
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.blocking_connection[577492] ERROR Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused'); first exception - None
2021-11-09 17:26:45 DellXPS-8BX90Z2 pika.adapters.blocking_connection[577492] ERROR Error in _create_connection().
Traceback (most recent call last):
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
Traceback (most recent call last):
  File "examples/server.py", line 47, in <module>
    server(workq, resultq)
  File "examples/server.py", line 14, in server
    work_queue.send({"id": i, "cmd": f'echo "{i}"'})
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/wipac_telemetry/tracing_tools/spans.py", line 228, in wrapper
    return func(*args, **kwargs)
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient/queue.py", line 129, in send
    self.raw_pub_queue.send_message(msg)
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient/queue.py", line 74, in raw_pub_queue
    self._pub_queue = self._backend.create_pub_queue(
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 357, in create_pub_queue
    q.connect()
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 81, in connect
    super().connect()
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/mqclient_rabbitmq/rabbitmq.py", line 45, in connect
    self.connection = pika.BlockingConnection(
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 360, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/home/eevans/repos/MQClient-RabbitMQ/.tox/py/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 451, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError


Opening folder opens blank buffer

Did you check the docs and existing issues?

[X] I have read the docs
[X] I have searched the existing issues

Neovim version (nvim -v)
9.4.1
Operating system/version
13.4
Describe the bug
When I open en folder with cr it opens a blank buffer
I am using lazy as plugin manager and just followed the docs
Steps To Reproduce
:Oil
Enter on any folder
Expected Behavior
Open folder and display contents
Directory structure
No response
Repro
-- save as repro.lua
-- run with nvim -u repro.lua
-- DO NOT change the paths
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "runtime", "cache" }) do
  vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "--single-branch",
    "https://github.com/folke/lazy.nvim.git",
    lazypath,
  })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  "folke/tokyonight.nvim",
  {
        "stevearc/oil.nvim",
        config = function()
            require("oil").setup({
              -- add any needed settings here
            })
        end,
  },
  -- add any other plugins here
}
require("lazy").setup(plugins, {
  root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

Did you check the bug with a clean config?

[X] I have confirmed that the bug reproduces with nvim -u repro.lua using the repro.lua file above.

Nevermind found, view_options, `show_hidden' was the culprit


Macro: R2, R3 and R4 connection points shown as R1, R2 and R3 on monomer preview tooltip for (R2,R3,R4) peptide

Steps to Reproduce

Switch to Macromolecules view
Load this file: (R2,R3,R4).zip
Hover mouse over peptide

Actual behavior

R2, R3 and R4 connection points shown as R1, R2 and R3 on monomer preview tooltip
Expected behavior
R2, R3 and R4 connection points shown on monomer preview tooltip
Desktop (please complete the following information):

OS: win10
Browser Chrome 120.0.6099.110 (Официальная сборка), (64 бит)

Ketcher version [e.g. v2.4.2].

Verified.


hamming: test requiring a function opposed to an object

For more consistency.
Most tests (at least the first few) require a function opposed to an object. There is no reason to break with this habit in this particular test.

Yeah, hamming doesn't need an object. Thanks! Would you mind squashing the commits, please?

Done :)

Thanks!


Build VIPSDISP on windows

Hi @jcupitt, is there any way I can build vipsdip on Windows? I successfully built it from source on ubuntu, and I tried to install everything I needed on MINGW64.  The problem I faced is in meson.build find_library 'm', so I bypass it using cc.find_library('m', required: false). But when I try to run ninja I got a lot of errors.
here a copy of part of errors:
FAILED: src/vipsdisp.exe.p/meson-generated_.._vipsdispmarshal.c.obj "cl" "-Isrc/vipsdisp.exe.p" "-Isrc" "-I../src" "-IC:/msys64/mingw64/include/libgsf-1" "-IC:/msys64/m ingw64/include/libxml2" "-IC:/msys64/mingw64/include/" "-IC:/msys64/mingw64/include/pango-1.0" "-IC: /msys64/mingw64/include/harfbuzz" "-IC:/msys64/mingw64/include/fribidi" "-IC:/msys64/mingw64/include /librsvg-2.0" "-IC:/msys64/mingw64/include/gdk-pixbuf-2.0" "-IC:/msys64/mingw64/include/webp" "-IC:/ msys64/mingw64/include/cairo" "-IC:/msys64/mingw64/include/freetype2" "-IC:/msys64/mingw64/include/l ibpng16" "-IC:/msys64/mingw64/include/pixman-1" "-IC:/msys64/mingw64/include/OpenEXR" "-IC:/msys64/m ingw64/include/Imath" "-IC:/msys64/mingw64/include/openjpeg-2.5" "-IC:/msys64/mingw64/include/orc-0. 4" "-IC:/msys64/mingw64/include/glib-2.0" "-IC:/msys64/mingw64/lib/glib-2.0/include" "-IC:/msys64/mi ngw64/include/gtk-4.0" "-IC:/msys64/mingw64/include/graphene-1.0" "-IC:/msys64/mingw64/lib/graphene- 1.0/include" "-I." "/MDd" "/nologo" "/showIncludes" "/utf-8" "/W2" "/std:c11" "/Od" "/Zi" "-DHAVE_CO NFIG_H" "-mfpmath=sse" "-msse" "-msse2" "-DLIBDEFLATE_DLL" "/Fdsrc/vipsdisp.exe.p/meson-generated_.. _vipsdispmarshal.c.pdb" /Fosrc/vipsdisp.exe.p/meson-generated_.._vipsdispmarshal.c.obj "/c" src/vips dispmarshal.c CreateProcess failed: The system cannot find the file specified.
Any suggestions?

Hi @euzada,
I would start with the libvips build system:
https://github.com/libvips/build-win64-mxe
And add the various gtk4 binaries.

The vipsdisp branch adds support for this. It seems to build successfully and works as intended.

To build locally, you can use:
$ git clone -b vipsdisp https://github.com/libvips/build-win64-mxe.git
$ cd build-win64-mxe/
$ ./build.sh web x86_64 shared --with-disp

Woah, that's fantastic Kleis! Nice job!
What's performance like? Can you zoom smoothly on large images?

Zooming and panning on large images also appear to be seamless; I think the performance is on par with that on Linux. See for example this screencast:
https://github.com/jcupitt/vipsdisp/assets/12746591/69f28944-8e8d-4e83-9de4-0af529d2d219
The only minor thing I noticed is that the icon for the previous and next page is broken.

I suppose that's a problem with the icon theme?
What if you hold down i? It's supposed to do a smooth 60fps zoom on the mouse pointer.

Yeah, it sounds like some sort of issue with the theme. Here's a screencast while holding down i.
https://github.com/jcupitt/vipsdisp/assets/12746591/896b8bc4-78f1-4a02-b0d0-d43dacb121f7

So ... hot. I'll try a a build here.

I noticed a tiny thing on ubuntu 23.04 -- I see:
$ ./build.sh web x86_64 shared --with-disp
Error: short-name "buildpack-deps:bullseye" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

I had to disable podman by editing build.sh.

What a good news. Thank you @kleisauke  and @jcupitt  for sharing. I will try to build it tomorrow on my windows and leave you feedback after building and testing.

The vipsdisp branch adds support for this. It seems to build successfully and works as intended.

To build locally, you can use:
$ git clone -b vipsdisp https://github.com/libvips/build-win64-mxe.git
$ cd build-win64-mxe/
$ ./build.sh web x86_64 shared --with-disp

Hi @kleisauke , is that branch on the top of 8.15.0 rc1 or 8.14?

is that branch on the top of 8.15.0 rc1 or 8.14?

It's based on top of v8.15.0-rc1, as required by vipsdisp v2.6.1.
https://github.com/jcupitt/vipsdisp/blob/35496349adf6463f4ea2ac73c0049a05edfe7dc9/meson.build#L40-L41

$ ./build.sh web x86_64 shared --with-disp
Error: short-name "buildpack-deps:bullseye" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

I had to disable podman by editing build.sh.

I think this patch would also fix that:
--- a/build.sh
+++ b/build.sh
@@ -186,7 +186,7 @@ fi
 mkdir -p $tmpdir

 # Ensure latest Debian stable base image
-$oci_runtime pull buildpack-deps:bullseye
+$oci_runtime pull docker.io/library/buildpack-deps:bullseye

 # Create a machine image with all the required build tools pre-installed
 $oci_runtime build -t libvips-build-win-mxe container
--- a/container/Dockerfile
+++ b/container/Dockerfile
@@ -1,4 +1,4 @@
-FROM buildpack-deps:bullseye
+FROM docker.io/library/buildpack-deps:bullseye

 RUN apt-get update \
   && apt-get install -y \

But indeed, perhaps I should disable that Podman in favor of Docker if-else statement, since Fedora/RHEL also have an RPM called podman-docker, which can be used as a drop-in placement for the docker utility.

The build was successful. It builds V8.15.0 rc1 and vipsdisp.exe. Loaded a very large image (7.5GB ) with no issues. The previous and next page icons are missing as mentioned previously.

We should make a list of small issues with the win build since we're all looking at it. I noticed:

missing icon theme
missing theme
no dbus.exe, leading to slow startup ... we could disable dbus integration (is it useful on win? I'm not sure) or include the dbus exe
drag and drop for file loading would be nice (nip2 on win has this, it was simple to support back then at least)
if you fullscreen, the right-click menu stops working (why?? how odd) and you have to press F11 to get back to windowed mode

Are there any others?

When I first open the executable I see terminal with:
(vipsdisp.exe:17092): GLib-GIO-WARNING **: 15:06:47.724: win32 session dbus binary not found
Than when I tried to save an image in another tiff format, I got:
(vipsdisp.exe:17092): GLib-GObject-CRITICAL **: 15:06:57.945: ../glib-2.78.1/gobject/gsignal.c:2451: signal id '0' is invalid for instance '00000262ee2848b0'

(vipsdisp.exe:17092): GLib-GObject-CRITICAL **: 15:06:57.948: ../glib-2.78.1/gobject/gsignal.c:2451: signal id '0' is invalid for instance '00000262ee2849f0'

(vipsdisp.exe:17092): GLib-GObject-CRITICAL **: 15:06:58.056: ../glib-2.78.1/gobject/gsignal.c:2677: instance '00000262ee2849f0' has no handler with id '0'

(vipsdisp.exe:17092): GLib-GObject-CRITICAL **: 15:06:58.059: ../glib-2.78.1/gobject/gsignal.c:2730: instance '00000262ee2849f0' has no handler with id '0'
Unknown type for property "keep"

I also noticed:

view-grid-symbolic is missing from GTK's fallback icon theme. Issue https://gitlab.gnome.org/GNOME/gtk/-/issues/6184.
GFileInfo created without standard::type errors when opening C:\ in the file chooser.

no dbus.exe, leading to slow startup ... we could disable dbus integration (is it useful on win? I'm not sure) or include the exe in the build (probably not worth it)

I've tentatively "fixed" this with commit https://github.com/libvips/build-win64-mxe/commit/6ad6917957f93b16e33b953c3e0a8d716ec2d8c4. I'm not sure if D-Bus integration can be disabled, it looks like a required thing.

(vipsdisp.exe:17092): GLib-GObject-CRITICAL **: 15:06:57.945: ../glib-2.78.1/gobject/gsignal.c:2451: signal id '0' is invalid for instance '00000262ee2848b0'

(vipsdisp.exe:17092): GLib-GObject-CRITICAL **: 15:06:57.948: ../glib-2.78.1/gobject/gsignal.c:2451: signal id '0' is invalid for instance '00000262ee2849f0'

(vipsdisp.exe:17092): GLib-GObject-CRITICAL **: 15:06:58.056: ../glib-2.78.1/gobject/gsignal.c:2677: instance '00000262ee2849f0' has no handler with id '0'

(vipsdisp.exe:17092): GLib-GObject-CRITICAL **: 15:06:58.059: ../glib-2.78.1/gobject/gsignal.c:2730: instance '00000262ee2849f0' has no handler with id '0'
Unknown type for property "keep"

I'm not sure about those gsignal errors, but the Unknown type for property "keep" warning can be fixed by adding a case for G_IS_PARAM_SPEC_FLAGS( pspec ) in save_options_add_option(), similar to G_IS_PARAM_SPEC_ENUM( pspec ).
https://github.com/jcupitt/vipsdisp/blob/02b5250c602d062b2908db987bd3ea381b890971/src/saveoptions.c#L377
(adding support for flags as save option would also make VipsForeignPngFilter work)

Ah you're right, the save-as dialog is missing some stuff, and doesn't resize too well. I have to make a long train journey today, I'll have a poke at it.

I fixed up sizing and added flags support, so that resolved the keep message, I think.

Save as TIFF  seems to work fine on linux, perhaps there's a win issue here?

I build the latest update. Now, the gdbus.exe is compiled and created in the bin folder and the first error (vipsdisp.exe:17092): GLib-GIO-WARNING **: 15:06:47.724: win32 session dbus binary not found is fixed.
The Unknown type for property "keep" is also fixed.
Saving the Tiff (or tif) is completed successfully with these information in terminal. but the save works.  Hope this is helpful.
(vipsdisp.exe:22024): GLib-GObject-CRITICAL **: 09:38:38.900: ../glib-2.78.1/gobject/gsignal.c:2451: signal id '0' is invalid for instance '0000022ea0be0a10'

(vipsdisp.exe:22024): GLib-GObject-CRITICAL **: 09:38:38.905: ../glib-2.78.1/gobject/gsignal.c:2451: signal id '0' is invalid for instance '0000022ea0be0910'

(vipsdisp.exe:22024): GLib-GObject-CRITICAL **: 09:38:38.996: ../glib-2.78.1/gobject/gsignal.c:2677: instance '0000022ea0be0910' has no handler with id '0'

(vipsdisp.exe:22024): GLib-GObject-CRITICAL **: 09:38:38.998: ../glib-2.78.1/gobject/gsignal.c:2730: instance '0000022ea0be0910' has no handler with id '0'

That's great! Is app startup faster? It was quite slow before. It ought to pop up a window almost instantly.
I don't know what's causing the signal errors, I don't see them here :( Some gtk/glib compat issue, perhaps?
i/o not being smooth can happen with some file formats. Are files that were smooth before hitching now?

That's great! Is app startup faster? It was quite slow before. It ought to pop up a window almost instantly.
I don't know what's causing the signal errors, I don't see them here :( Some gtk/glib compat issue, perhaps?
i/o not being smooth can happen with some file formats. Are files that were smooth before hitching now?

I check the previous version, the i/o where not smooth. So maybe it is my file size. Will generate smaller file and test again.
Is there any way to load it without opening Terminal first?

Windows .exe files have a special flag inside for "I'm a GUI app". If this is not set, Windows will automatically make a terminal and display program input and output.
We just need to set a compiler flag when linking vipsdisp.exe ... I forget what it's called.

Ah, setting -mwindows when linking vipsdisp.exe, I think. We might also need to define WinMain(), I'm not sure.

It looks like the specifying win_subsystem: 'windows' in Meson's executable() function would fix that.
https://mesonbuild.com/Reference-manual_functions.html#executable_win_subsystem
https://github.com/GNOME/gtk/blob/020ef51cb00fdf0e4a440468d41ed8256881da17/demos/widget-factory/meson.build#L71
I'll try a patch and also investigate those signal errors.

That's great! Is app startup faster? It was quite slow before. It ought to pop up a window almost instantly.

I did some test using the following command in bash for windows and did not see any difference in the latest version.
time "E:\vips-dev-8.15\bin\vipsdisp.exe" | ps ax | grep vipsdisp | grep -v grep | awk '{print $1}' | xargs kill
I don't know if I should run a different command.
The output was something like that for both versions:
real    0m1.080s
user    0m0.015s
sys     0m0.321s

Oh, clever idea. Perhaps it's just slow to load all those DLLs? There are a lot of them. A static build might be faster.

I build it with the following command:
./build.sh web x86_64 shared --with-disp
Do I try static build?

Yes, try static.
I tried this in vipsdispapp.c:
static void
vipsdisp_app_activate( GApplication *app )
{
    ImageWindow *win;

    win = image_window_new( VIPSDISP_APP( app ) );
    gtk_window_present( GTK_WINDOW( win ) );
    process_events();
    exit(0);
}

ie. on startup, show the main window, process a few events, then quit. On ubuntu I see:
$ time vipsdisp

real	0m0.200s
user	0m0.160s
sys	0m0.075s

which feels fast.

Yes, try static.
I tried this in vipsdispapp.c:
static void
vipsdisp_app_activate( GApplication *app )
{
    ImageWindow *win;

    win = image_window_new( VIPSDISP_APP( app ) );
    gtk_window_present( GTK_WINDOW( win ) );
    process_events();
    exit(0);
}

ie. on startup, show the main window, process a few events, then quit. On ubuntu I see:
$ time vipsdisp

real	0m0.200s
user	0m0.160s
sys	0m0.075s

which feels fast.

I am getting this error trying to build it with static flag like that:   ./build.sh web x86_64 static --with-disp
ERROR: GTK cannot be built as a statically linked library on Windows.

You need to build libvips static, then vipsdisp dynamic.

MR https://gitlab.gnome.org/GNOME/glib/-/merge_requests/3678 will fix the signal errors.
PR https://github.com/jcupitt/vipsdisp/pull/25 will cause Windows' native open/save dialog to open instead of GTK's built-in one.

The vipsdisp branch has been updated to incorporate this. I'll have a look at the disabled static build and GUI app hint tomorrow.

It looks like specifying win_subsystem: 'windows' in Meson's executable() function would fix that.

Oh nice find, I added that to src/meson.build

Great! I can confirm that commit ff7b8cb76134226af8cfb14a28c7af30dda68b75 removes the stray terminal window.
To avoid having to spend any extra CPU time building these binaries, I've uploaded the 64-bit libvips 8.15.0-rc1 Windows binaries with vipsdisp here:
https://libvips-packaging.s3.amazonaws.com/vips-dev-w64-web-8.15.0-rc1-disp.zip
https://libvips-packaging.s3.amazonaws.com/vips-dev-w64-all-8.15.0-rc1-disp.zip
(these binaries were built against commit https://github.com/libvips/build-win64-mxe/commit/9c86ee10cf999711eb69178c16c96ce3c52c1680)

I can confirm no Terminal at the start. and Save windows use Windows explorer. I will do some test for the speed later today.
Thank you.

I tried out "loupe", the new gtk4 image viewer that gnome have made:
https://gitlab.gnome.org/GNOME/loupe
With st-francis.jpg:
$ vipsheader st-francis.jpg
st-francis.jpg: 30000x26319 uchar, 3 bands, srgb, jpegload

The version in ubuntu 23.10 loads the image in 13.5s and needs 7.5gb of memory. Panning needs maybe 8% of CPU on my PC.
The Gnome nightly build loads in 13.5s again, but only needs 2.3gb of memory. CPU use seems about the same.
On this ubuntu 23.10 PC, a release build of vipsdisp loads the image in 8s and needs 1.3gb of memory. Panning is abou 15% of CPU.
So we win on load speed and memory use, as you'd hope heh. We could probably get CPU use down --- at the moment, vipsdisp culls tiles to the viewport every frame, but we only really need to cull on tile boundaries.

... I tuned the zoom speed and initial window size to match loupe.

I don't know if on windows it is the same. When I try to zoom on my PC, The GPU is the one how manage the zoom and increased to almost 10% while zooming (I believe it is good things) . Do you see any GPU increase on Linux?

Yes, it uses the GPU on linux too.

I start a fresh build. I am not sure it will apply to the loupe integration. It will take time to build llvm.
I use git config pull.rebase true and it said:
Updating 7681c50a..c185fad1
Fast-forward
 docs/build-matrix.html | 6 +++---
 docs/packages.json     | 6 +++---
 src/freetds.mk         | 4 ++--
 src/libjpeg-turbo.mk   | 4 ++--
 src/openssl.mk         | 4 ++--
 5 files changed, 12 insertions(+), 12 deletions(-)

Do you know if this will apply to loupe integration?

There's no loupe integration, I was just benchmarking it for comparison.

Oh, you mean the changes to vipsdisp to match loupe? You'd need to change vipsdisp.mk to build git master.

Yes, thgis is what I meant. I test the last build and the load takes 70ms to start the app. I used Measure-Command {.\vipsdisp.exe} in powershell.

if you fullscreen, the right-click menu stops working (why?? how odd) and you have to press F11 to get back to windowed mode

In Full screen with F11, the menu will be hidden behind the GUI in the background. If you do "Windows + d" to see the desktop than do it again "Win + d" the menu will appear on the front. I believe the solution is to force the right-click menu to always be on the top.

In Full screen with F11, the menu will be hidden behind the GUI in the background. If you do "Windows + d" to see the desktop than do it again "Win + d" the menu will appear on the front.

Ah, interesting!
gtk doesn't have keep_above:
https://docs.gtk.org/gtk4/class.Window.html
vipsdisp uses the same menu for the burger button on the header bar, and right-click on the image. Maybe this is the cause? Perhaps we ought to use separate menus, or remove the right-click menu?

I think this is all done now.


Added required concat method for Audio files

Q
A

Bug fix?
yes

New feature?
no

BC breaks?
no

Deprecations?
no

Fixed tickets

Related issues/PRs
#502

License
MIT

What's in this PR?
Added the concat method to the FFMpeg\Media\Audio which was missing.
Why?
Concatenation of audio files does not work without it.

I added tests. I used two audio files with CC license and added attribution to the readme as the license requires.

Both adjustments made, how does it look now?

Just one more thing and we're done. :) Thanks for your patience.

No worries, this package has been really useful for me. Last adjustment done. I was trying to imitate the existing tests.

Thanks, I'll get back to this as soon as I found some time :)

I have been successfully using my dev branch in production for months now, so this seems to be working just fine. Any ideas when then this could me merged and release?

@lasselehtinen Sorry I missed your branch, but thanks for pinging me. If the update does not fail, I'm happy to merge and release.


Feedback: Don't put the hcloud token in a var

It is not up to you how the consumer sets up the module and the hcloud provider, They may source it through a env var to avoid having it in the state. If it is a var it will be in the state!

This cannot currently be realized because the token must be written to the hcloud secret and we therefore need access to it.


stdlib._net_http_ResponseWriter is not seen as an http.Hijacker

The following program sample.go triggers an unexpected result
package main

import (
	"bufio"
	"errors"
	"log"
	"net"
	"net/http"
	"net/http/httputil"
	"net/url"
)

type Middleware struct {
	next http.Handler
}

func (h *Middleware) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
	wrw := &WrappedWriter{writer: rw}
	h.next.ServeHTTP(wrw, r)
}

func main() {
	// forwards to e.g. websocat -s 6060
	dest, err := url.Parse("http://localhost:6060")
	if err != nil {
		log.Fatal(err)
	}
	next := httputil.NewSingleHostReverseProxy(dest)
	http.Handle("/", &Middleware{next})
	log.Fatal(http.ListenAndServe(":8080", nil))
}

type WrappedWriter struct {
	writer http.ResponseWriter
}

func (w *WrappedWriter) Header() http.Header {
	return w.writer.Header()
}

func (w *WrappedWriter) Write(buf []byte) (int, error) {
	return w.writer.Write(buf)
}

func (w *WrappedWriter) WriteHeader(statusCode int) {
	w.writer.WriteHeader(statusCode)
}

func (w *WrappedWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) {
	h, ok := w.writer.(http.Hijacker)
	if !ok {
		return nil, nil, errors.New("hijack not supported")
	}
	return h.Hijack()
}

Expected result
As a backend, run e.g. (from https://github.com/vi/websocat , there's a brew for it):
$ websocat -s 6060

Start the above wrapped reverse proxy:
$ go run ./sample.go

Issue a websocket request on the reverse proxy (which will trigger the hijacking):
$ websocat ws://<IP_ADDRESS>:8080/
hello
("hello" should be printed on the backend output)

Got
$ yaegi run ./sample.go

$ websocat ws://<IP_ADDRESS>:8080/
websocat: WebSocketError: Received unexpected status code (502 Bad Gateway)
websocat: error running

Error on the yaegi side:
reverseproxy.go:490: http: proxy error: can't switch protocols using non-Hijacker ResponseWriter type stdlib._net_http_ResponseWriter

Yaegi Version
cb642c44bad588189feb00760b5f0ec4ff5cdada (devel)
Additional Notes
Related traefik issue:
https://github.com/traefik/traefik/issues/8682

Another example, from the same root cause, but with a different impact, and in a different part of the stdlib.
Realistically, in this case it means yaegi probably wouldn't break the big picture behaviour (if the user implementation of WriteTo was honest and not wonky like mine), but it definitely would break some potential optimizations.
package main

import (
	"io"
	"log"
	"os"
	"strings"
)

type WrappedReader struct {
	reader io.Reader
}

func (wr WrappedReader) Read(p []byte) (n int, err error) {
	return wr.reader.Read(p)
}

// Of course, this implementation is completely stupid because it does not write
// to the intended writer, as any honest WriteTo implementation should. its
// implemtion is just to make obvious the divergence of behaviour with yaegi.
func (wr WrappedReader) WriteTo(w io.Writer) (n int64, err error) {
	// Ignore w, send to Stdout to prove whether this WriteTo is used.
	data, err := io.ReadAll(wr)
	if err != nil {
		return 0, err
	}
	nn, err := os.Stdout.Write(data)
	return int64(nn), err
}

func main() {
	f := strings.NewReader("hello world")
	wr := WrappedReader{reader: f}

	// behind the scenes, io.Copy is supposed to use wr.WriteTo if the implementation exists.
	// With Go, it works as expected, i.e. the output is sent to os.Stdout.
	// With Yaegi, it doesn't, i.e. the output is sent to io.Discard.
	if _, err := io.Copy(io.Discard, wr); err != nil {
		log.Fatal(err)
	}
}

Another different/interesting case:
package main

import (
	"image"
	"io"
	"log"
	"os"
)

type WrappedReader struct {
	reader io.Reader
}

func (wr WrappedReader) Read(p []byte) (n int, err error) {
	return wr.reader.Read(p)
}

func (wr WrappedReader) Peek(i int) ([]byte, error) {
	println("BLABLA")
	return []byte("hello"), nil
}

func main() {
	f := strings.NewReader("hello world")
	wr := WrappedReader{reader: f}
	_, format, err := image.Decode(wr)
	if err != nil {
		log.Fatal(err)
	}
	println(format)
}

this one is a little bit different, because the implemented interface is actually a non-exported one this time, in image/format.go:
// A reader is an io.Reader that can also peek ahead.
type reader interface {
	io.Reader
	Peek(int) ([]byte, error)
}

which is used in e.g. asReader, which is called in Decode.
As with the other cases above, the Go compiler sees the interface is being implemented, whereas Yaegi does not.


Swipe calender for next/prev

Hi there,
I am implementing ng-fullcalendar on my Ionic app.
I want to add option to navigate next/prev by swiping right/left.
How can I add this functionality ?
Thank you.

Hi! use https://github.com/HsuanXyz/ion2-calendar for ionic,
or use hammerjs
(swipeleft)
(swiperight)


types(NODE-3729): add id type to TSchema return types

Description
What is changing?
This PR simply adds the _id field whenever TSchema/Document is returned as a type by using the existing WithId type, as explained in the linked issue
Is there new documentation needed for these changes?
No, because this was already the expected behavior and nothing changed code-wise
What is the motivation for this change?
While working on a personal project of mine, I had to delete the _id property from the findOne result object in order to compare it to a similarly generated object, and noticed I was getting a TS error saying the property _id didn't exist on the returned type. This PR aims to fix that so that the _id type is always present regardless of whether the Collection type has it or not. Jira issue: https://jira.mongodb.org/browse/NODE-3729

Double check the following

[ ] Ran npm run check:lint script
[X] Self-review completed using the steps outlined here
[X] PR title follows the correct format: <type>(NODE-xxxx)<!>: <description>
[X] Changes are covered by tests
[X] New TODOs have a related JIRA ticket

⚠ Note: I was not able to run the test scripts as I kept getting an error related to node-gyp when using npm i, npm ci, or yarn, so if someone else could run them I'd highly appreciate it! (Alternatively, a CI could be added to run that)

Found more types that need to be changed, will commit in a bit

Think that's all, please let me know if I missed any/if I changed some that I shouldn't have

@ImRodry Hi there, thank you for reaching out.
Unfortunately, we had an issue with our CI builds that blocked all our tests from running and necessitated an update to main - if you could, please rebase your branch onto the new main to ensure the CI is able to run on your PR.
As far as your actual suggested changes - you do bring up a good point that something like a findOne will by default return the document with an _id field in addition to the fields specified in the collection type. However, we must be careful when applying this logic to aggregation pipelines, because the result of an aggregation operation almost never has the shape of the source document. I think it is a good idea to add WithId to our defaults for find operations where we specify TSchema as the default return type, but we need to make sure that we do not impose this on aggregation results, which need to accept a separate type definition wherever they are expected.

Absolutely! Will rebase in a second
as for the aggregation pipelines, I’m gonna be honest: I’m not too familiar with those so it’s possible I made some mistakes. Please review the lines you think should be reverted/changed so that I can apply those changes

@ImRodry Thanks! We'll take a closer look soon.
With find specifically, if you do something like collection.find({}, { projection: { _id: 0 } }), you will get all documents without the _id property; and, in general, you can use the projection option to include or exclude fields from the return result, which is why we allow an optional override type on find operations, so that you are able to specify the return type you expect given your specific set of options.
I mentioned aggregation because I see you also modified the aggregation cursor file: the AggregationCursor is used by the aggregate api in the driver and operates on a "pipeline" of transformation stages, where the end result depends on what transformations you specify (e.g., you could sum all values of a particular field in matching documents, and return a document with a single "total" field) - you can read more here, if curious.

Thanks for the projection tip! I was actually not aware of that and it highly helps my code.
What I did in the AggregationCursor file was mostly by looking at it and trying to understand what types it was expecting, so I'll be awaiting the full review once you have time for that :D
Also slight note: the CI is still failing and there is no CI to run the test script, is that intentional? I still cannot install node_modules locally after the rebase so I'm not sure what to do

I somehow managed to run the test script and it's giving me errors on two files: src/change_stream.ts:714:64 and src/change_stream:339:43 that I am not sure how to fix. If you wish I can simply revert the changes to change_stream and leave the others that I am more familiar with.
I also noticed an error in your test script in build:ts as it's missing the keyword node after the &&

I also noticed an error in your test script in build:ts as it's missing the keyword node after the &&

It makes use of the relative path to the installed version of tsc our driver depends on ./node_modules/typescript/bin/tsc. (due to this issue: https://github.com/SamVerschueren/tsd/issues/122)
It should be marked as executable so no need to proceed it node. 🤔  Maybe there's some difference in environment? Windows OS perhaps?

It should be marked as executable so no need to proceed it node. 🤔 Maybe there's some difference in environment? Windows OS perhaps?

Well yeah I am on windows and had to add the node keyword before that path. It executed just fine after this however, maybe it should be added to work on all OS's

The specific use case surrounding our findOne and find functions has been covered by #3039
@ImRodry this may help your use case, is there further changes still needed to get the other APIs working as you would expect?

I only touched the other APIs because I felt the PR would be incomplete without doing so, but I have personally never messed with those. I will see if that PR fixes my issue and if it doesn’t, I’ll open a new one. For now, I’ll close this. Thanks!