[13:08] <Guest4913> hey folks! just out of curiosity is there something going on with ppa.launchpad.net at the moment?  we are looking at some timeouts starting from 07:00 UTC today connecting to ppa.launchpad.net (<IP_ADDRESS>) from AWS us-east-1 (Northern Virginia/USA), this seems to be a transient issue and we have assembled a lot of different signals to debug this on our side at this point. happy to put some relevant information wherever it can be useful to y'all,
[13:13] <cjwatson> Guest4913: #canonical-sysadmin might know better.  From this point of view ppa.launchpad.net is basically just an Apache server and the Launchpad team doesn't manage the networking.  I don't know of any issues though
[13:15] <Guest4913> cjwatson: thank you for the answer!! i will ask over there if something's going on
[15:09] <NIXKnight> Hi
[15:10] <NIXKnight> is there a problem with launchpad right nmow
[15:10] <NIXKnight> we are having a lot of timeouts
[15:11] <NIXKnight> for some repos
[15:11] <NIXKnight> that we are pulling from on AWS.
[15:16] <cjwatson> NIXKnight: Need to be more specific.  Are you talking about ppa.launchpad.net?
[15:16] <NIXKnight> cjwatson yes
[15:17] <cjwatson> NIXKnight: #canonical-sysadmin manages that - from LP staff's point of view it's just an Apache server and we don't control the networking.  Somebody else reported some problems from AWS earlier today as well and filed a ticket
[15:25] <adzuci> Thanks cjwatson, I saw the message from https://irclogs.ubuntu.com/2021/08/10/%23launchpad.txt but don't have the history from @canonical-sysadmin, do you have the link to the ticket handy?
[15:26] <cjwatson> adzuci: Well, I mean I have https://rt.ubuntu.com/Ticket/Display.html?id=36786 but even I can't see it at the moment
[15:26] <cjwatson> So not sure how helpful that is
[15:26] <cjwatson> The only reports I've seen so far are from AWS
[15:28] <NIXKnight> Thanks cjwatson
[15:28] <adzuci> Thanks for the ticket, NIXKnight and I have also only seen this issue in AWS and were wondering if it could be related to rate limiting logic.
[15:28] <cjwatson> I'm not aware of rate-limiting at our end
[15:29] <cjwatson> Which is not to say there isn't any
[15:29] <cjwatson> If it were rate-limiting then you'd normally expect an immediate 503 rather than a timeout, though
[17:08] <timo_> Hello all, I'm having issues uploading some files. Both through the web interface and Python launchpadlib. At first glance it might be a filesize issue, a 4.6 tar.gz file uploads fine, while a 36 mb zip and 74 mb dmg return a 502 error. Full error is here: https://pastebin.com/W4MinBRA
[17:09] <timo_> Note that similar files uploaded fine in the past, this is just a version update of my application. Am I hitting a size limit for my project perhaps?
[17:10] <cjwatson> timo_: There are no project-specific size limits; I don't immediately know what the problem might be here, and am about to finish up for the day.  Could you please file a support ticket using the link in the topic?
[17:11] <timo_> Sure, what's the right place? https://answers.launchpad.net/launchpad ?
[17:12] <cjwatson> timo_: Yes
[17:12] <cjwatson> I haven't so far found any obvious diagnostics on our side
[17:12] <cjwatson> Is it consistently reproducible?
[17:13] <timo_> Yes, over the last 4-5 days.
[17:15] <cjwatson> timo_: Details of the URL you're attempting to upload to would likely be helpful (for searching logs)
[17:16] <timo_> Here: https://launchpad.net/pigeonplanner/4.2/4.1.1
[17:20] <cjwatson> 2021-08-10 15:22:48.295Z INFO talisker.wsgi "POST /1.0/pigeonplanner/4.2/4.1.1" method=POST path=/1.0/pigeonplanner/4.2/4.1.1 status=500 duration_ms=29827.591 ip=<IP_ADDRESS> proto=HTTP/1.1 length=14 request_length=36708708 request_type="[...]" forwarded=[...] ua="[...]" logging_count=6 logging_time_ms=0.951 nonpython_actions=1 scheme=https host=api.launchpad.net ...
[17:20] <cjwatson> ... request_id=75401959-8d56-42e9-9ff1-27f580f4c5d6
[17:20] <cjwatson> Looks like an application timeout, not yet sure why
[17:22] <cjwatson> https://oops.canonical.com/oops/?oopsid=OOPS-0206f5d07d58d3b361bd01f8fb50e617 (Canonical-internal link, just posting here for reference)
[17:23] <timo_> Thanks for checking.
[17:24] <cjwatson> https://docs.gunicorn.org/en/stable/settings.html#timeout defaults to 30 seconds and we don't override that.  I wonder if we need to bump that so that it's larger than all the per-page hard timeouts we've explicitly set
[17:25] <cjwatson> timo_: Could you post the launchpadlib code you're using (feel free to redact anything private), just so that I can make sure I'm looking in the right place at our end?
[17:26] <cjwatson> We switched to gunicorn fairly recently, so this may be correlated with that
[17:26] <cjwatson> I don't know why small files would be a problem though, that certainly seems backwards
[17:26] <cjwatson> Anyway, dinnertime
[17:29] <timo_> Note that the small 4.1mb file uploaded just fine, it's the two others that fail. Also through the web interface, so it's not limited to using launchpadlib.
[17:30] <timo_> It looks like the last time I uploadsed files was a year ago, so recent changes could make sense.
[17:30] <timo_> Anyway, thanks for looking at it last minute. Highly appreciated.
[20:19] <cjwatson> timo_: Oh, I misread your "4.6 tar.gz" line as "4.6G tar.gz" for some reason
[20:20] <cjwatson> No wonder I was confused
[20:23] <cjwatson> timo_: (Having that support ticket would still be good though - we have a lot going on and chances are I'll forget otherwise.)
[20:57] <timo_> cjwatson: my bad, I didn't realise I missed the "mb" bit.
[20:58] <timo_> I'll create a ticket tomorrow with all details. Thanks again.
[21:46] <cjwatson> The nova API service in bos02 has fallen over, so we can't reset builder VMs.  I'm going to bed shortly but have alerted IS