[00:23] <lifeless> -> pet store and stuff
[00:24] <lifeless> gary-lunch: look at rev 12564 in devel to see how to do a with query with my patched storm
[00:37] <wgrant> I tried to create a packageupload(archive, distroseries, status) index on DF. It works for queries like archive=534, distroseries=X, status=Y. But when archive=1 it refuses to use the index. I guess it thinks its stats know better.
[00:38] <lifeless> do you have a sample query ?
[00:39] <wgrant> EXPLAIN ANALYZE SELECT PackageUpload.archive, PackageUpload.changesfile, PackageUpload.date_created, PackageUpload.distroseries, PackageUpload.id, PackageUpload.pocket, PackageUpload.signing_key, PackageUpload.status FROM PackageUpload WHERE packageupload.distroseries = 103 AND packageupload.archive = 534 AND packageupload.status IN (0) ORDER BY PackageUpload.id DESC LIMIT 31 OFFSET 0;
[00:39] <wgrant> That uses the index.
[00:39] <wgrant> And is fast.
[00:39] <wgrant> s/534/1/, no index, slow.
[00:39] <lifeless> gotta run, bbiab
[00:50] <wgrant> Oh. Now that is interesting.
[00:51] <wgrant> It uses the index if distroseries=102
[00:51] <wgrant> But not 103.
[00:53] <mwhudson> is it the usual thing where it thinks it's going to be looking at most of the table anyway?
[00:53] <wgrant> I presume so.
[00:54] <wgrant> 103 does show up in pg_stats, so that may be it. But other archives that show up there don't see the same issue, so I might have to check out pg_statistics to see what it's doing.
[00:54] <wgrant> Or harass stub.
[01:17] <gary-lunch> cool thanks lifeless
[01:17]  * gary-lunch wonders how he became gary-lunch
[01:18] <StevenK> gary-lunch: It's a bit late for lunch, isn't it?
[01:18] <gary_poster> yes indeed
[01:18] <StevenK> Almost lunch time here, in fact.
[01:19] <wgrant> Ah.
[01:19] <wgrant> It will happily use packageupload(archive, distroseries, status, id).
[01:20] <wgrant> I guess it must have decided that archive=1, distroseries=103 meant the final sort would be too expensive.
[01:20] <wgrant> Despite the fact that there are actually 0 rows.
[01:25] <lifeless> wgrant:  Limit  (cost=0.00..954.05 rows=31 width=36) (actual time=263.988..31328.809 rows=1 loops=1)
[01:25] <lifeless>    ->  Index Scan Backward using distroreleasequeue_pkey on packageupload  (cost=0.00..93404.35 rows=3035 width=36) (actual time=263.985..31328.804 rows=1 loops=1)
[01:25] <lifeless>          Filter: ((distroseries = 103) AND (archive = 1) AND (status = 0))
[01:25] <lifeless>  Total runtime: 31328.871 ms
[01:25] <lifeless> (4 rows)
[01:25] <lifeless> 1.5 seconds hot
[01:25] <lifeless> wgrant: thats qastaging
[01:26] <wgrant> lifeless: Right, that's about what I see on mawson.
[01:26] <wgrant> Except about 25% faster.
[01:26] <wgrant> I'll talk to stub about unbreaking the indices tonight.
[01:27] <wgrant> Since the current ones were designed before archives were introduced...
[01:27] <lifeless> you're adding an index to match the critera more closely
[01:27] <lifeless> whats the explain analyze of the query that (isn't using) the new index look like
[01:28] <lifeless> wgrant: can you drop the order by ?
[01:28] <lifeless> Time: 37.729 ms
[01:28] <wgrant> lifeless: The current indices are not reasonable.
[01:28] <wgrant> We need the order by.
[01:28] <lifeless> wgrant: why?
[01:28] <wgrant> packageupload(distroseries, status) does not make sense any more.
[01:28] <wgrant> Since you never query PUs by series outside an archive.
[01:29] <lifeless> wgrant: sure, I'm not debating that, just looking at whats going on
[01:29] <lifeless> wgrant: I meant 'why do we need the order by'
[01:29] <wgrant> lifeless: So we don't show them in arbitrary order on DistroSeries:+queue
[01:30] <lifeless> wgrant: why would that matter ?
[01:30] <wgrant> Because that would make batching sort of bad.
[01:32] <lifeless> wgrant:  SELECT PackageUpload.archive, PackageUpload.changesfile, PackageUpload.date_created, PackageUpload.distroseries, PackageUpload.id, PackageUpload.pocket, PackageUpload.signing_key, PackageUpload.status FROM
[01:32] <lifeless>                 PackageUpload WHERE packageupload.distroseries = 103 AND packageupload.archive = 1 AND packageupload.status IN (0) ORDER BY packageupload.distroseries, packageupload.status, PackageUpload.date_created desc LIMIT 31 OFFSET 0;
[01:33] <wgrant> lifeless: That is cheating.
[01:33] <lifeless> wgrant: so is life
[01:33] <wgrant> There's no index on PackageUpload.date_created, so it has no choice.
[01:33] <wgrant> It will be slow if it returns lots of results.
[01:34] <lifeless> wgrant: right, so we would not want to do that for every query
[01:34] <lifeless> wgrant: but the queue isn't unlimited
[01:34] <lifeless> wgrant: and lots could be a few thousand without perf worries
[01:34] <lifeless> wgrant: I agree we need to improve the indices
[01:35] <wgrant> lifeless: https://launchpad.net/ubuntu/lucid/+queue?queue_state=3&queue_text=
[01:35] <wgrant> 130000
[01:36] <lifeless> wgrant: whats 3
[01:36] <wgrant> lifeless: Done.
[01:36] <lifeless> we should delete them
[01:36] <wgrant> You could argue that.
[01:36] <lifeless> s/c/sh/
[01:37] <wgrant> It requires reworking lots of Soyuz. We should probably do that, but that is a *huge* *huge* thing.
[01:37] <wgrant> We need to develop a strategy.
[01:37] <LPCIBot> Project windmill build #37: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/37/
[01:37] <wgrant> On how we handle history like this.
[01:38] <wgrant> And apply it everywhere.
[01:38] <StevenK> Windmill, you make me very sad.
[01:39] <wgrant> StevenK: It's right, though.
[01:40] <lifeless> wgrant: so, does the queue *post* need new indices to be fixed?
[01:40] <lifeless> wgrant: or would my hack give breathing space?
[01:40] <wgrant> lifeless: I personally think that some of this data is useful. The table is small, and it's extremely fast to query with the right indices.
[01:40] <wgrant> lifeless: I don't think that hack is useful.
[01:40] <wgrant> It would work.
[01:40] <lifeless> ok
[01:40] <wgrant> But an index is less of a hack, and works everywhere.
[01:56] <lifeless> 455 Time Outs
[01:56] <lifeless> nice
[01:57] <lifeless> bugger
[01:57] <lifeless>       23 /   22  POFile:+translate
[01:57] <lifeless> 20 /    0  LanguageSet:CollectionResource:#languages
[01:58] <wgrant> Wow.
[01:58] <lifeless> ah
[01:58] <wgrant> Bugger?
[01:58] <lifeless> wgrant: things we had fixed
[01:58] <wgrant> They weren't deployed yet, were they?
[01:58] <lifeless> but yeah
[01:58] <lifeless> 9 hours of them
[01:58] <wgrant> They are now.
[01:58] <lifeless> tomorrow will give better results again
[01:58] <wgrant> Yep.
[01:59] <wgrant> This is why I want 12-hourly ones :(
[01:59] <lifeless> wgrant: I want a sliding window
[01:59] <wgrant> Right.
[01:59] <lifeless> plus per-deploy
[01:59] <wgrant> But that needs oops-repository.
[01:59] <wgrant> (I suspect)
[01:59] <lifeless> well, can be done in the current code, just a matter of effort
[02:00] <wgrant> Right.
[02:00] <lifeless> I'm going to look into hbase as an alternative to cassandra
[02:00] <wgrant> Oh?
[02:00] <lifeless> need to send a raft of emails around
[02:00] <wgrant> I haven't heard much about hbase...
[02:00] <wgrant> But it's Apache too, right?
[02:00] <lifeless> that aspect is fine
[02:01] <lifeless> the issue with cassandra is lack of range queries
[02:01] <wgrant> :(
[02:10] <wgrant> lifeless: Do we expect daemons to survive DB restarts? I know eg. librarian doesn't.
[02:11] <lifeless> wgrant: yes
[02:11] <lifeless> wgrant: exact implementation to be figured out later
[02:11] <wgrant> So that's a no?
[02:12] <lifeless> today, restarting the db may wed daemons. If so thats a bug.
[02:12] <wgrant> I'm tempted to leave poppy broken in that situation, given that most other daemons will have the same issue.
[02:14] <wgrant> Can you see any problem with that?
[02:15] <wgrant> It's not going to try to hold transactions open forever any more, so it shouldn't get killed :)
[02:15] <lifeless> if you know of a specific defect, please make sure there is a high bug about it
[02:16] <wgrant> I think it's a general defect in all daemons...
[02:16] <lifeless> wgrant: I wouldn't assume that
[02:18] <wgrant> Anyway, lunch, double poppy cowboy fixing, and then +copy-packages UI optimisations.
[02:18] <wgrant> Because the actual copy operation is now fast :(
[02:18] <lifeless> \o/
[02:19] <StevenK> wgrant: That timeout was due to UI?
[02:24] <LPCIBot> Project windmill build #38: STILL FAILING in 46 min: https://hudson.wedontsleep.org/job/windmill/38/
[02:26] <wgrant> StevenK: Yes ;(
[02:28] <wgrant> At least I think so.
[02:29] <wgrant> It's a bit hard to tell exactly what's what.
[02:29] <wgrant> Because the UI stuff generates a lot of queries.
[02:30] <wgrant> It's also not copying binaries, which is a case that is less optimised. But it doesn't seem to get to the actual copies.
[02:30] <wgrant> Erm.
[02:30] <wgrant> "I get a timeout error when trying to copy a few (5+) packages from a ppa"
[02:30] <wgrant> But field.selected_sources has at least 15...
[02:31] <wgrant> I suppose that is technically still 5+...
[02:31] <lifeless> :>
[02:32] <lifeless> wow
[02:32] <lifeless> Aggregate  (cost=7238627.72..7238627.73
[02:32] <wgrant> Which query's that?
[02:33] <lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=1895B1870
[02:33] <wgrant> Ah.
[02:35] <lifeless> partitioning the query into two unioned on each fti column gets
[02:35] <lifeless>  Aggregate  (cost=291.55..291.56 rows=1 width=0)
[02:36] <lifeless> I've been dreading this
[02:36] <lifeless> but its time to bite the bullet and do it
[02:36] <lifeless> I'll do my administrivia stuff monday
[02:37] <wgrant> Heh
[02:37] <lifeless> I wonder
[02:37] <lifeless> maybe I'll just ditch the bugtask fti column
[02:37] <wgrant> Does that just index targetnamecache?
[02:40] <lifeless> 16 more hits
[02:40] <lifeless> for synaptics touchpad
[02:40] <lifeless> targetnamecache and statusexplanation
[02:40] <lifeless> -wtf-
[02:41] <wgrant> I thought statusexplanation was actually dead now.
[02:41] <wgrant> Apparently not :/
[02:42] <lifeless> COMMENT ON COLUMN BugTask.statusexplanation IS 'A place to store bug task specific information as free text';
[02:42] <wgrant> You know what it is, right?
[02:42] <lifeless> no idea
[02:43] <wgrant> You know how the old task edit form had a comment field?
[02:43] <wgrant> That adds a comment and sets statusexplanation.
[02:43] <wgrant> Those changes are also logged in bugactivity.
[02:43] <lifeless> lib/lp/bugs/scripts/bugzilla.py uses it
[02:43] <wgrant> If that's what I think it is, it will die.
[02:44] <wgrant> I wonder if it still works.
[02:44] <lifeless> the task edit form still exists
[02:44] <lifeless> browser/bugtask.py has
[02:44] <wgrant> Yes, but it doesn't need to set statusexplanation.
[02:44] <lifeless> if comment_on_change:
[02:44] <lifeless> bugtask.statusexplanation = comment_on_change
[02:45] <lifeless>  select count(*) from bugtask where statusexplanation is null or statusexplanation='';
[02:45] <lifeless>  count
[02:45] <lifeless> --------
[02:45] <lifeless>  531321
[02:45] <lifeless> select count(*) from bugtask where statusexplanation is not null;
[02:45] <lifeless>  count
[02:45] <lifeless> --------
[02:45] <lifeless>  480008
[02:45] <lifeless> bah
[02:45] <wgrant> Hmmm?
[02:45] <lifeless> select count(*) from bugtask where statusexplanation is not null and statusexplanation != '';
[02:45] <lifeless>  count
[02:45] <lifeless> --------
[02:45] <lifeless> so 300K set, 530K unset
[02:45] <lifeless>  300198
[02:45] <wgrant> Right. Kill it.
[02:46] <wgrant> It has been thoroughly deprecated for more than 5 years.
[02:46] <wgrant> We have these wonderful things called comments to store that information now :)
[02:49] <lifeless> bug/88545
[02:50] <lifeless> bug 88545
[02:50] <lifeless> no mup?
[02:52] <thumper> statusexplanation?
[02:52] <lifeless> a discarded idea
[02:57] <StevenK> Is it actually used in the codebase at all?
[02:58] <lifeless> grep ftw
[02:58] <wgrant> The only place I've seen it exposed is BugTask:+activity
[02:59] <StevenK> lifeless: Isn't there another clean up you wanted to do with BugComment and Bug<name escapes me currently>
[02:59] <lifeless> jcsackett is on that
[02:59] <lifeless> or at least on nearby stuff
[02:59] <wgrant> BugMessage.visible was moved this morning.
[03:00] <StevenK> Ah yes BugComment versus BugMessage
[03:00] <lifeless> there is a completely redundant python class constructed from bugmessage+message
[03:00] <lifeless> if we had separation of concerns - a persistence layer - it would make sense
[03:00] <lifeless> but its purely a Data Object at the moment, valueless.
[03:01] <StevenK> libwebkitgtk-1.0.0-dbg DIALCF!
[03:02] <wgrant> StevenK: Why?
[03:02] <StevenK> 240MiB debug packages make me cranky.
[03:02] <wgrant> Hah.
[03:03] <StevenK> And it's Webkit, which I have other issues with.
[03:06] <lifeless> like, it existing?
[03:07] <StevenK> lifeless: Is that your issue with it?
[03:07] <lifeless> no
[03:07] <lifeless> bundled libraries
[03:07] <lifeless> bugs
[03:07] <lifeless> footprint
[03:07] <StevenK> That's one of my issues -- my major gripe of it is just how much crap is in it
[03:07] <wgrant> I prefer WebKit to Gecko... at least once it has been deGoogled.
[03:08] <lifeless> its a whole desktop
[03:08] <wgrant> ie. had the hundred external libraries excised.
[03:08] <lifeless> what could be wrong
[03:08] <StevenK> Which gives us wonderful things like the aforementioned 240MiB debug packages.
[03:08] <StevenK> wgrant: Oh, Gecko is no better
[03:08] <StevenK> From the same people who gave us building UIs from XML
[03:08] <StevenK> Oh crap, now I need to review my lunch.
[03:08] <wgrant> Um, most GTK apps now build UIs from XML :)
[03:09] <lifeless> wgrant: exactly
[03:09] <StevenK> Didn't XUL start that madness?
[03:09] <wgrant> It is unclear that it is madness.
[03:10] <lifeless> wgrant: you saw https://code.launchpad.net/~lifeless/storm/with-without-datetime right ?
[03:10] <StevenK> Anyway, yay for feeping creaturism
[03:10] <wgrant> lifeless: I never looked at the code.
[03:10] <lifeless> wgrant: thats fine
[03:11] <lifeless> whats the script that gives committer frequencies in lp
[03:12] <wgrant> What do you mean? Number of commits by person for a particular branch?
[03:12] <lifeless> within a timeframe, or per month, or whatever.
[03:12] <lifeless> yes
[03:13] <lifeless> I can has review? https://code.launchpad.net/~lifeless/launchpad/bug-716774/+merge/52966
[03:13] <wgrant> I don't know of anything beyond bzr-stats...
[03:14] <lifeless> wgrant: does PoppyFileWriter's validateGPG affect sftp uploads?
[03:14] <wgrant> lifeless: No.
[03:14] <wgrant> It's only used in FTP for now.
[03:14] <wgrant> (WTF, I know, but convenient)
[03:15] <lifeless> thanks
[03:15] <wgrant> I guess I should start that IR at some point.
[03:16] <lifeless> or ask me to do it
[03:16] <lifeless> but don't sit on it
[03:16] <wgrant> I'm sure you have better things to do. I'll do it.
[03:20] <StevenK> wgrant: Julian says he couldn't work out how to do it for SFTP uploads. Which made me sad. :_(
[03:20] <wgrant> :(
[03:24] <StevenK> I'm hoping poppy-{s,}ftp now get a good hard refactoring
[03:24] <StevenK> $DEITY knows they deserve it.
[03:24] <wgrant> A renaming wouldn't go astray either.
[03:26] <StevenK> Oh, and death to zope ftp would be nice, too.
[03:26] <wgrant> lifeless: An advertised feature is no longer working. Not Critical?
[03:28] <lifeless> wgrant: we've never had it live
[03:28] <lifeless> wgrant: we tried and it went boom
[03:28] <wgrant> It was working fine for like an hour :P
[03:28] <lifeless> wgrant: that doesn't count
[03:29] <wgrant> :(
[03:29] <lifeless> julian blogging about it was great enthusiasm, but we probably want to wait for things to be stable before announcing
[03:29] <wgrant> Indeed.
[03:30] <lifeless> wgrant: point is, if we stopped work on the improvement right now, we haven't actually gone back on anything folk could have started depending on.
[03:30] <lifeless> that said, I think we should push forward and get it going
[03:30] <lifeless> it will make a huge difference to usability.
[03:31] <lifeless> and that will mean happier and less confused users
[03:31] <lifeless> lower support overhead and easier training for new ubuntu devs
[03:36] <wgrant> thumper: Hi.
[03:36] <thumper> wgrant: hi
[03:36] <wgrant> thumper: Could you mentor https://code.launchpad.net/~lifeless/launchpad/bug-716774/+merge/52966?
[03:37]  * thumper looks
[03:37] <wgrant> Anyone want to review https://code.launchpad.net/~wgrant/launchpad/poppy-transaction-unbreakage/+merge/52967?
[03:39] <StevenK> wgrant: Done. thumper ^
[03:40] <thumper> OMG
[03:40] <thumper> my recursive query worked
[03:40] <thumper> http://pastebin.ubuntu.com/578642/
[03:41] <StevenK> According to the postgres docs it's iteration, not recursion :_)
[03:41] <thumper> does postgres have stored procs?
[03:42] <StevenK> From memory, yes
[03:43] <thumper> hmm
[03:43] <StevenK> wgrant: Wait. Are you sure you ran bzr add?
[03:43] <thumper> doing the blueprint dependencies in the db has to be better than doing X queries
[03:47] <lifeless> thumper: yes
[03:47] <lifeless> thumper: you can do that with the storm we have in trunk
[03:47] <thumper> lifeless: how?
[03:47] <lifeless> store.with(SQL('RECURSIVE dependencies(id, name, depth) AS (
[03:47] <lifeless>         SELECT s.id, s.name, 1
[03:47] <lifeless>         FROM specification s
[03:47] <lifeless>         where s.id = 20428
[03:47] <lifeless>       UNION ALL
[03:48] <lifeless>         SELECT s.id, s.name, d.depth + 1
[03:48] <lifeless>         FROM specificationdependency sd, dependencies d, specification s
[03:48] <lifeless>         WHERE sd.specification = d.id
[03:48] <lifeless>           AND s.id = sd.dependency
[03:48] <lifeless> )')
[03:48] <lifeless> bah
[03:48] <lifeless> store.with_
[03:48] <lifeless> so store.with_(clause).find(Specification)
[03:48] <lifeless> or whatever
[03:48] <lifeless> probably
[03:49] <lifeless> so store.with_(clause).find(Specification, SQL('Specification.id in (select id from dependencies))')
[03:51] <lifeless> thumper: care to review https://code.launchpad.net/~lifeless/launchpad/bug-716774/+merge/52966 ?
[03:52] <thumper> lifeless: I have
[03:52] <lifeless> oh sweet
[03:52] <lifeless> thanks
[03:52] <lifeless> sorry I hadn't noticed
[03:52]  * lifeless ec2inates
[03:52] <StevenK> Which just sounds ... wrong.
[03:55] <wgrant> StevenK: What did I miss?
[03:55] <StevenK> wgrant: It looks to me that you moved all the imports but not the actual method itself?
[03:55] <wgrant> 211	[03:55] <wgrant> 212	[03:56] <StevenK> Ah ha. Objection withdrawn, the MP is still approved.
[03:58] <thumper> I'm actually pretty stoked about that blueprint dependency query
[03:58] <thumper> now to squeeze it into the code somehow
[03:59] <wgrant> Let's hope we don't have to revert to pg 8.3 :)
[03:59] <StevenK> Haha
[04:00] <thumper> wgrant: recursive queries were added in 8.3
[04:01] <thumper> at least I thought so
[04:01] <StevenK> WITH RECURSIVE was added in 8.4
[04:01] <wgrant> I thought WITH was added in 8.4.
[04:01] <thumper> ah...
[04:01] <wgrant> Right.
[04:01] <thumper> I wonder what the likelyhood of that is then...
[04:01] <wgrant> 0
[04:01] <thumper> \o/
[04:08] <StevenK> thumper: Can haz mentor? :-)
[04:08] <wgrant> thumper: Could you mentor https://code.launchpad.net/~wgrant/launchpad/poppy-transaction-unbreakage/+merge/52967?
[04:08] <StevenK> Haha
[04:08]  * thumper looks
[04:09] <wgrant> Thanks.
[04:09] <thumper> 'tis past wine o'clock on a Friday
[04:09] <thumper> you're lucky :0
[04:09] <lifeless> we're not reverting
[04:09] <lifeless> not to 8.3
[04:09] <lifeless> that goose has flown
[04:10] <StevenK> Can we revert to 9.0? That sounds like FUN.
[04:11] <thumper> done
[04:11] <wgrant> 9.0 already? Well done.
[04:12] <StevenK> Damn it, I just missed a devel build on buildbot
[04:12] <StevenK> ... by about 30 seconds. :-(
[04:12] <wgrant> We can't roll out until Tuesday, either :(
[04:12] <lifeless> tuesday?
[04:13] <wgrant> ACT public holiday on Monday, and no mthaddon next week.
[04:13] <StevenK> Haha, awesome
[04:15]  * thumper wines a little
[04:15]  * thumper AFK
[04:15] <StevenK> s/w/wh/
[04:16]  * thumper pokes StevenK in the eye
[04:16] <StevenK> Ow!
[04:50] <wgrant> :(
[04:51] <wgrant> Does BugTask.status serve any useful purpose?
[05:09] <lifeless> uhm
[05:09] <lifeless> its where we store triaged etc
[05:11] <wgrant> Opinion/Invalid/Won'tFix aren't usefully separated, New is implicit, Confirmed is implied by affects, In Progress is effectively implied by assignee, Fix Committed and Fix Released are implied by MP status.
[05:11] <wgrant> So we can replace them all with "valid" and "incomplete" flags.
[05:12] <cody-somerville> or customizable bug statuses. \o/
[05:12] <StevenK> wgrant: Right, which works for *us*, but may not work for other projects.
[05:13] <wgrant> StevenK: They work for anyone that uses MPs.
[05:13] <wgrant> Unless they are abusing statuses, in which case they are wrong :)
[05:13] <cody-somerville> no
[05:13] <cody-somerville> its not 'wrong'. its the system speaking to you.
[05:14] <StevenK> cody-somerville: What possible need do you have for customizing bug statuses?
[05:14] <cody-somerville> Oh, OEM Services has tons since we abuse bugs for other things.
[05:14] <huwshimi> wgrant: Take a look at the statuses proposed on this page: https://dev.launchpad.net/IssueTracker
[05:15] <wgrant> huwshimi: Mmm. Better, but still not optimal.
[05:16] <cody-somerville> ugh. not better IMHO.
[05:16] <StevenK> What, two statuses? Fixed and Unfixed? :-P
[05:17] <wgrant> cody-somerville: Why?
[05:17] <wgrant> The only significant change is merging Fix Committed and Fix Released.
[05:17] <StevenK> I think cody-somerville wants *more* statuses, not less.
[05:20] <lifeless> so workflow around bugs is a fascinating topic
[05:23] <cody-somerville> indeed
[05:25] <StevenK> OH FER. libwebkitgtk-1.0-0-dbg isn't enough, libwebkitgtk-3.0-0-dbg exists too
[05:26] <wgrant> StevenK: Why are you dealing with them?
[05:26] <StevenK> Local mirror sync
[05:27] <wgrant> Ah.
[05:28] <wgrant> :(
[05:28] <wgrant> DF doesn't have memcached installed.
[05:29] <cody-somerville> but for what its worth... I think the current set of bug statuses are alright. I'm much more interested and would get much more value  (professionally and personally) from things like build-from-branch and derivative distributions. :)
[05:30] <StevenK> cody-somerville: You know recipe builds are "done", right?
[05:30] <wgrant> Recipes != BFB
[05:30] <StevenK> Oh, right.
[05:30] <StevenK> That saddens me.
[05:31] <StevenK> lol. OO.o is in NBS
[05:33] <cody-somerville> cause although tweaking the bug statuses might make LP better to some degree, it isn't going to make it more competitive against say github or OBS.
[05:34] <cody-somerville> Anyhow, I should try and get some sleep. :) *waves*
[05:49]  * wgrant murders get(Build)StatusSummar(y|ies)(For(Builds|Source(Publication|IdsAndArchive)))
[05:52] <StevenK> wgrant: Seriously?
[05:53] <LPCIBot> Project windmill build #39: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/39/
[05:59] <wgrant> StevenK: Well, bits of them at least.
[06:00] <StevenK> Aww
[06:00] <wgrant> SourceForge's new platform is interesting.
[06:00] <wgrant> Rather GitHubish.
[06:01] <wgrant> And far less awful than their old one, at first glance.
[06:07] <huwshimi> wgrant: They've just open sourced their backend too
[06:07] <wgrant> Yeah, that's the new one.
[06:07] <wgrant> Currently grabbing it...
[06:08] <StevenK> Haha, of course you are.
[06:11] <wallyworld> wgrant: i can't seem to run any windmill tests locally (to fix the breakage). is there a config item i need to tweak do you know?
[06:14] <wgrant> wallyworld: --layer=WindmillLayer
[06:15] <wgrant> It defaults to --layer=!(MailmanLayer|WindmillLayer)
[06:15] <wallyworld> wgrant: thanks
[06:16] <wallyworld> was that what you changed to stop the tests by default?
[06:16] <wgrant> Yeah. It was previously just !MailmanLayer
[06:17] <wgrant> (see buildout-templates/bin/test.in)
[06:32] <wgrant> wallyworld: I considered just changing the vocab keys to series.name, but I'm not sure how much the form machinery will like that.
[06:33] <wgrant> Evening stub.
[06:34] <wallyworld> wgrant: you mean to fix the test? i'm still considering how to fix it. shouldn't be too hard
[06:34] <wgrant> wallyworld: The test isn't broken.
[06:34] <wgrant> wallyworld: The code is.
[06:34] <stub> yo
[06:35] <wallyworld> wgrant: yeah, can see that :-(
[06:35] <wgrant> stub: mawson's garbo-hourly is a bit unhappy. Do I have to manually apply session.sql?
[06:36] <lifeless> stub: how hard is it to update fti ?
[06:36] <stub> Parts of launchpad_session.sql
[06:36] <stub> wgrant: What bit is it barfing on?
[06:36] <wgrant>  -> http://librarian.dogfood.launchpad.net/57720639/nqvma824KMUojGISQIkXotVXlmD.txt (function cursor_fetch(unknown, integer) does not exist
[06:36] <stub> lifeless: Just edit the data structure at the top of fti.py - it handles creating all the triggers, columns, indexes etc. automatically
[06:37] <lifeless> stub: can we update it live?
[06:37] <stub> lifeless: No. Adding or changing the triggers requires a table lock.
[06:37] <lifeless> k
[06:39] <lifeless> was thinking of sneaking up on https://code.launchpad.net/bugs/88545
[06:39] <stub> wgrant: So from database/schema/launchpad_session.sql you want CREATE OR REPLACE FUNCTION cursor_fetch and the GRANT EXECUTE ON FUNCTION cursor_fetch
[06:40] <stub> (I'd applied this manually to staging and production systems before the code landed but forgot about mawson)
[06:41] <stub> wgrant: That needs to be applied to the session database on mawson of course
[06:41] <wgrant> Right.
[06:41] <wgrant> Thanks.
[06:42] <wgrant> Much happier, thanks.
[06:43] <wgrant> stub: I also have some concerns about PackageUpload's indices.
[06:46] <stub> wgrant: ?
[06:47] <wgrant> stub: See the last comment in bug #276950
[06:47] <wgrant> I don't think packageupload(distroseries, status) has been very useful in the last ~4 years.
[06:53] <stub> wgrant: It seems to be used, but for what I can't tell: http://paste.ubuntu.com/578697/
[06:54] <stub> Looks like we haven't deleted any GPG keys for a while though
[06:55] <wgrant> stub: I guess it might try to use that if there's no order, even if the archive is constrained.
[06:55] <wgrant> It avoids (archive, distroseries, status) for some values if it has to order by ID.
[06:55] <wgrant> But will use it for others.
[06:55] <StevenK> stub: Does that mean the signing_key index is unused?
[06:56] <stub> StevenK: It means no lookups are done using signingkey. That index exists to support deleting a row from the gpgkey table (which needs to locate any referencing rows).
[06:57] <StevenK> stub: Ah right, which also explains your deletion of GPG keys comment.
[06:58] <StevenK> I'm pondering a new key, my current one is 7.5 years old
[06:59] <stub> wgrant: Do you have a problem with the table, such as really slow restore times, disk space utilization, or slow insert times?
[06:59] <lifeless> slow queries
[06:59] <wgrant> stub: Just slow queries.
[06:59] <wgrant> I think the existing index is pointless.
[06:59] <wgrant> And should be replaced.
[06:59] <wgrant> There's an example query in the bug I referenced.
[06:59] <stub> It might make sense to replace it.
[07:00] <stub> poke mup
[07:00] <wgrant> It's been dead most of the day.
[07:00] <wgrant> https://bugs.launchpad.net/launchpad/+bug/276950
[07:02] <stub> wgrant: Your suggestion sounds sane. I wouldn't normally make such a specific index without knowing for sure it will be used, but you have already done that research.
[07:02] <wgrant> (archive, distroseries, status) queries are done extensively.
[07:03] <stub> We don't have to drop the existing index though - if it becomes unused I'll pick it up next time I do a scan. It will slow down inserts, but won't slow down selects.
[07:03] <stub> Right. I don't think they used to, because we didn't used to have ppas etc.
[07:03] <wgrant> Exactly.
[07:03] <wgrant> I think this index is just a few years out of date.
[07:04] <stub> Back from when the table was called DistroReleaseQueue it seems
[07:04] <stub> (maybe I can rename primary key indexes without pain now in 8.4?)
[07:04] <lifeless> stub: does postgresql have a log-slow-queries facility ?
[07:04] <wgrant> (I initially tried without id, but the slow view's queries sort by id, and in some conditions it reverts back to the plain id index)
[07:04] <stub> lifeless: yes
[07:05] <lifeless> stub: can we turn that on, set to 15000ms, and get some reports around it ? Ideally we'd tie it back to the request/script
[07:05] <lifeless> stub: I'm thinking with all the improvements we've done, we're at the point where starting to track slow queries isn't just insane
[07:05] <wgrant> lifeless: You clearly haven't seen the scripts lately.
[07:06] <lifeless> wgrant: I'm very worried about script performance
[07:07] <stub> log_min_duration_statement. We should enable it for limited users though, perhaps just one appserver, to keep the logs sane.
[07:07] <lifeless> stub: by sane, do you mean 'not overwhelming' ?
[07:07] <stub> Yes
[07:08] <lifeless> stub: rather than do it for limited users, lets set a higher minimum
[07:08] <lifeless> stub: that way we don't have to keep updating it as more users are added
[07:08] <stub> lifeless: So it will be full of garbo queries then.
[07:09] <stub> We expect some longer queries for the batch jobs - that is why they are batch jobs.
[07:09] <lifeless> stub: I want the backend tasks to be fast too
[07:10] <lifeless> stub: for a few reasons - latency, total overhead, mvcc garbage overhead
[07:10] <lifeless> stub: so I'm just as interested in garbo, rosetta etc as I am in appservers
[07:10] <stub> So you won't see any appserver queries then, because I'd need to set the timeout to something nuts like 10seconds.
[07:10] <lifeless> stub: I was suggesting an initial setting of 15 seconds
[07:10] <lifeless> stub: we see 15 second queries on appservers at the moment
[07:11] <lifeless> well, 13 now - cause they hit the timeout
[07:11] <stub> I'll turn it on at 15s and see how our logs survive.
[07:12] <lifeless> and expire on the server - but we still have a couple of overrides
[07:12] <lifeless> cool
[07:13] <stub> We already get the statements killed by timeout btw.  The format is bogus.
[07:14] <lifeless> the oops reporting system should be telling us about all of the expired ones
[07:14] <lifeless> and i see this pattern a lot:
[07:14] <lifeless> a few cheap queries
[07:14] <lifeless> a mammoth query
[07:14] <lifeless> a few cheap queries
[07:14] <lifeless> an expired query
[07:15] <stub> pgfouine might make a reasonable report even if we don't feed it full logs.
[07:18] <wgrant> lifeless: Bugzilla integration? You mean the one-off script that imported Ubuntu Bugzilla?
[07:19] <lifeless> wgrant: poppy could run in autocommit mode perhaps
[07:19] <lifeless> wgrant: grep for the field, you tell me
[07:21] <stub> I'm not seeing any log spam at 15s
[07:21] <wgrant> Wait 'til the publisher runs :)
 2011-03-11 07:22:14 UTC LOG:  duration: 88923.523 ms  statement: SELECT COUNT(*) FROM (SELECT DISTINCT BinaryPackageName.id, BinaryPackageName.name FROM BinaryPackageName, BinaryPackagePublishingHistory, BinaryPackageRelease, DistroArchSeries WHERE
[07:22] <stub>                     BinaryPackagePublishingHistory.binarypackagerelease =
[07:22] <stub>                         BinaryPackageRelease.id AND
[07:22] <stub>                     BinaryPackageRelease.binarypackagename =
[07:22] <stub>                         BinaryPackageName.id AND
[07:22] <stub>                     BinaryPackagePublishingHistory.status IN (1, 2) AND
[07:22] <stub>                     BinaryPackagePublishingHistory.pocket = 0 AND
[07:22] <stub>                     BinaryPackagePublishingHistory.distroarchseries =
[07:22] <stub>                         DistroArchSeries.id AND
[07:22] <stub>                     DistroArchSeries.distroseries = 103 AND
[07:22] <stub>                     BinaryPackagePublishingHistory.archive IN (1, 534)
[07:23] <stub>                      AND (1=1)) AS "_tmp"
[07:23] <stub> So its working.
[07:23] <stub> And hopefully the publisher doesn't have any queries that evil ;)
[07:23] <wgrant> Well, some of its queries take 10 minutes on mawson... they're not quite that bad on prod, though.
[07:23] <lifeless> stub: I'd leave it for a day
[07:24] <lifeless> review on monday
[07:24] <stub> I'm about to poke the GSAs to twiddle logrotate to guard against this taking us down.
[07:24] <lifeless> stub: awesome, thanks
[08:23] <adeuring> good morning
[08:41] <henninge> Hi adeuring!
[08:41] <adeuring> moin henninge
[08:44] <henninge> adeuring: I am almost done with the check list.
[08:45] <adeuring> great
[09:03] <henninge> adeuring: pulled, merged, pushed
[09:03] <adeuring> cool
[09:03] <henninge> And it looks really cool, too.
[09:04] <henninge> and works well for most items because the views I am linking to use "ReturnToReferrerMixin", so I mostly get back to the details page.
[09:04] <henninge> only the last one doesn't
[09:05] <henninge> adeuring: I have to run an errand, be back later.
[09:19] <jam> lifeless: can you ec2land https://code.launchpad.net/~jameinel/launchpad/loggerhead-732481/+merge/52902
[09:19] <jam> or maybe adeuring ^^
[09:20] <adeuring> jam: yes, I'll land it.
[09:20] <jam> adeuring: thanks.
[10:07] <LPCIBot> Project windmill build #40: STILL FAILING in 1 hr 8 min: https://hudson.wedontsleep.org/job/windmill/40/
[11:08] <lifeless> jam: something that would be nice: a patch to the loggerhead Makefile so that 'make check' runs the test suite with the loggerhead tree from .
[12:02] <deryck> Morning, all.
[12:08] <allenap> deryck: Good morning :)
[12:09] <allenap> deryck: I have a branch that does some work on Javascript building. As an expert in this area, would you have time to review it?
[12:09] <deryck> allenap, sure.
[12:09] <allenap> deryck: https://code.launchpad.net/~allenap/launchpad/alt-javascript-build/+merge/53006
[12:09] <allenap> Thank you :)
[12:10] <deryck> np
[12:10] <deryck> looking now....
[12:20] <jtv> Anyone here really familiar with feature flags?  I just can't figure out what the proper way to handle an "enabled" setting is.  Should I set true/false, or on/off?  Should I evaluate its value as boolean, or compare to my chosen "on" string?  How do I clear the flag in a test, and how in production?
[12:20] <jtv> The existing code seems divided on the subject.
[12:20] <wgrant> jtv: I standardised it a few weeks back...
[12:20] <wgrant> It should be pretty standard, except for memcache.
[12:20] <wgrant> if getFeatureFlag('foo'):
[12:20] <jtv> And then what are appropriate values for that flag?
[12:21] <wgrant> It's not optimal ('' = False, everything else True), but it's simple and easy to use in TAL.
[12:21] <wgrant> We agreed that it was the least evil solution for now.
[12:21] <jtv> Can we set a feature flag to the empty string>
[12:21] <jtv> ?
[12:21] <wgrant> Yes.
[12:22] <deryck> allenap, looks great, thanks for doing this!  r=me
[12:22] <allenap> deryck: Woohoo :) Thank you.
[12:22] <deryck> np :-)
[12:25] <jtv> wgrant: oh well, it seems to satisfy my tests.  Thanks.
[12:25] <jtv> And then I guess I can use any string whatsoever to indicate "on": yes, true, enable, on, or even off, disable, die, nothanks, samovar, …
[12:26] <wgrant> Yes :(
[12:27] <jtv> Is the data still entered as a tab-separated text field?  Because that gives extra juice to the whole "blank" thing.
[12:27] <wgrant> The separator can be any whitespace.
[12:31] <allenap> Is it me, or is Launchpad very slow right now?
[12:32] <bigjools> allenap: I don't know, are you slow?
[12:32] <allenap> bigjools: Fairly slow, yes.
[12:32] <wgrant> It's slow, but not slower than usual AFAICT.
[12:33] <allenap> Okay, thanks. It's not doing anything at all for me now.
[12:34] <bigjools> might be a bad appserver
[12:37] <allenap> bigjools: Sounds about right... it's okay again now.
[12:37] <jtv> wgrant: the feature flags documentation still uses "off" as an example.
[12:40] <wgrant> jtv: Where?
[12:40] <wgrant> jtv: The docs on /+feature-info should be up to date.
[12:40] <jtv> wgrant: https://dev.launchpad.net/FeatureFlags
[12:41] <wgrant> jtv: Fixed.
[12:41] <jtv> great, thanks
[12:42] <jtv> And the "on|off" bit as well, I see.
[12:47] <jtv> adeuring: got a review for you!  https://code.launchpad.net/~jtv/launchpad/bug-733132/+merge/53009
[12:47] <adeuring> jtv: I'll look
[12:47] <jtv> dankeschön
[13:12] <adeuring> jtv: the branch looks good. only one detail: FEATURE_FLAG as a variable name does not tell what the feature is about. Could you rename it to something like ENABLE_DERIVED_SERIES_JOBS ?
[13:12] <jtv> adeuring: I'm in two minds about that… doesn't the module it's in do enough to disambiguate it?
[13:12] <jtv> I could import it into the test as distroseriesdifferencejob.FEATURE_FLAG
[13:13] <adeuring> jtv: well, we could have two feature flags in this module
[13:13] <jtv> That way the name is always disambiguated.
[13:13] <jtv> Ah I see what you mean.
[13:14] <adeuring> I prefer names which tell what is done over how things are done
[13:15] <jtv> Okay, then how about ENABLING_FEATURE_FLAG?
[13:15] <jtv> Because it is important to my mind that it is the feature flag, not e.g. a boolean that controls the feature.
[13:15] <jtv> If the test refers to it as distroseriesdifferencejob.ENABLING_FEATURE_FLAG, I think that'd be about as clear as it could be.
[13:16] <adeuring> jtv: what about FEATURE_FLAG_ENABLE_DERIVED_JOBS?
[13:16] <jtv> Gets a bit long, and basically for the purpose of repeating the module name.
[13:16] <adeuring> without say what is enabled, it looks to me still like a label on an electrial switch just saying "SWITCH" ;)
[13:18] <jtv> Well that would make sense in the context of a little circuit board when you know what the board is for, just not what outside part should be attached to the particular piece of wire you're looking at.
[13:18] <adeuring> jtv, ok, so... ENABLE_MODULE? or FLAG_ENALBE_MODULE?
[13:19] <jtv> It would have to say "flag," but I don't see why it should say "module."
[13:19] <adeuring> well, I still think that its good to indicate what is enabled
[13:21] <jtv> Yes, but I'm relying on the "distroseriesdifferencejob" to imply that it's DistroSeriesDifferenceJob that's being enabled.
[13:22] <jtv> I don't think it'll confuse anyone, especially since it's not even being exported—it's just the test that refers to it from the outside.
[13:23] <adeuring> jtv: ok, for the test distroseriesdifferencejob.WHAETEVER is more or less fine, but _inside_ the module something like FEATURE_FLAG_ENABLE_MODULE would still make more sense to me
[13:25] <jtv> OK, I can't say I like it much but you have the advantage of a fresh outside look.  I'll use that.
[13:26] <adeuring> jtv: thanks :)
[13:26] <adeuring> so, r=me
[13:26] <jtv> Thanks
[13:35] <bigjools> gary_poster: another quick buildout question - if we have sourcecode deps in the dists tree but not mentioned in versions.cfg, are they used somewhere else or are they safe to remove?
[13:36] <gary_poster> maybe used somewhere else bigjools.  sadly, sourcecode is a separate mechanism
[13:36] <gary_poster> (and on call :-) )
[13:36] <bigjools> gary_poster: darn :/  ok thanks
[13:46] <allenap> adeuring: Hi Abel. Could you review a tiny branch for me please? https://code.launchpad.net/~allenap/launchpad/fix-librarian-startup-warnings/+merge/53018
[13:48] <adeuring> allenap: sure, I'll look
[13:48] <allenap> Thanks.
[13:53] <adeuring> allenap: r=me
[13:53] <allenap> adeuring: Cheers.
[13:57] <wgrant> le
[13:58] <wgrant> sinzui: Hi.
[13:58] <sinzui> hi wgrant
[13:58] <wgrant> sinzui: You upgraded lazr.restful to 0.17.4 a couple of days ago, right?
[13:58] <sinzui> I did
[13:58] <wgrant> 0.17.2 has a regression in exception handling.
[13:58] <wgrant> See #launchpad.
[13:59] <sinzui> was that fixed in 17.3 by leonardr
[13:59] <wgrant> No.
[13:59] <wgrant> It's still broken in 0.17.4 (which is on prod now)
[14:00] <wgrant> OOPS-1896STAGING128
[14:00] <sinzui> Well I am not going to fix that regression by restoring another regression
[14:00] <sinzui> I will need to talk to leonardr
[14:00] <wgrant> Indeed, that's what I was going to suggest.
[14:01] <wgrant> I haven't filed a bug, since it is 1am.
[14:01] <wgrant> Can you poke leonard about this when he appears?
[14:01] <sinzui> My change was to revert the xhtml-in-json because it was being injected into pages without being escaped
[14:01] <wgrant> Right.
[14:02] <wgrant> It just pulled in the exception handling changes too.
[14:02] <sinzui> 0.17.3 does something interesting with exception handling. It fixed some broken tests I saw in 0.17.2
[14:02] <wgrant> Ahhh.
[14:02] <wgrant> I think I see what's going on, maybe.
[14:03] <sinzui> Reverting that change breaks the test suite :)
[14:03] <wgrant> Hmm, no, I don't see it.
[14:03] <sinzui> Which was also why I was not going to land my work...I hand never seen my tests run successfully.
[14:03] <wgrant> But the 0.17.2 diff looks like a hack :/
[14:05] <wgrant> It also looks like it should be using .response_code, not .status...
[14:06] <LPCIBot> Project windmill build #41: STILL FAILING in 1 hr 8 min: https://hudson.wedontsleep.org/job/windmill/41/
[14:07] <sinzui> wgrant: I recall "status" reappeared in response output when I added this version to Lp. I assumed it was because of my change, not leonardr's
[14:08] <wgrant> Anyway, leonardr will save the world.
[14:09] <sinzui> wgrant: are you certain 0.17.2 was live? the expose(exception, code) signature was very new. like late last week
[14:10] <wgrant> sinzui: We rolled out your change (with 0.17.4) this morning, AFAIK.
[14:10] <sinzui> I had to update my deps to resolve the sugnature change
[14:11] <sinzui> right, and 0.17.3 was never released. But was 0.17.2 really the version in production. 0.17.1 looks like the version I have last week
[14:11] <wgrant> Hmm? It broke in 0.17.2, and is still broken in 0.17.4
[14:12] <wgrant> Timeouts OOPS like that staging OOPS I gave above.
[14:13] <wgrant> 0.17.4 was introduced in r12565, and r12568 is now on production.
[14:14] <abentley> deryck: bzr+ssh://bazaar.launchpad.net/~abentley/launchpad/js-translation
[14:16] <wgrant> Anyway, night.
[14:17] <abentley> deryck: http://pastebin.ubuntu.com/578842/
[14:20] <deryck> abentley, ci = new CheckItem({identifier: 'id1'})
[14:49] <jml> how do I search for bugs that don't have certain tags?
[14:51] <james_w> jml, https://bugs.launchpad.net/launchpad/+bug/81575
[14:51] <_mup_> Bug #81575: no way to search for absence of a tag <bugtag> <lp-bugs> <oem-services> <ubuntu-qa> <Launchpad itself:Fix Released by allenap> < https://launchpad.net/bugs/81575 >
[14:54] <jml> james_w: ta
[15:02] <jcsackett> is it bad that i'm excited a db patch i landed took under 10 minutes?
[15:03]  * jcsackett figures it probably is.
[15:03] <sinzui> Not bad
[15:03] <sinzui> will your patch break everything?
[15:05]  * jml trying out new battery.
[15:10] <jcsackett> sinzui: no, my patch fixes messages.
[15:10] <jcsackett> sinzui: it was the visibility db patch.
[15:11] <jcsackett> sinzui: real question is how long it takes on the weekend super-rebuild.
[15:51] <rvba> jcsackett: quick question: how did you know how long your patch took to be applied?
[15:55] <rvba> sinzui: thanks for your review ... I think I misunderstood the word 'resubmit' and did put a red mark on my own proposal ;-).
[16:05] <jcsackett> does staging have different timeout thresholds from production?
[16:06] <sinzui> rvba: I'll follow up the review when the diff completes its update
[16:16] <deryck> henninge, still around?
[16:17] <henninge> deryck: yes, for the moment
[16:17] <deryck> henninge, need just a quick ack that this looks sane for displaying "not translated" -- http://people.canonical.com/~deryck/in-upstream-msg.png
[16:17] <deryck> henninge, in your opinion.  not asking for a formal ui review ;)
[16:18] <henninge> that looks good, although we use "(not translated yet)" in other places  AFAIR
[16:21] <deryck> henninge, ok, I'll match that.  thanks.
[16:21] <bac> you still around adeuring?
[16:21] <adeuring> bac: yes
[16:22] <sinzui> rvba: r=me for code and tests
[16:22] <bac> adeuring: sorry it took me so long to show up today
[16:22] <adeuring> well, for an hour or so
[16:22] <adeuring> bac:  no problem :)
[16:23] <bac> adeuring: i'll take wallyworld's branch
[16:23] <adeuring> bac: cool, thanks
[16:24] <adeuring> bac; I must admit that I simply cducked out from looking at the loggerhead stuff -- I have no real clue about it...
[16:24] <bac> adeuring: you're the best judge of what you can do a good job reviewing
[16:25] <rvba> sinzui: nice. thx for the careful review. It's precious to me because I can find many layers of different ways to do the same things in the code.
[16:26] <sinzui> rvba: yes. We are very bad to removing code/test styles that we decide were bad
[16:27] <rvba> sinzui: I'll have to correct my own changes from last week.
[17:22] <deryck> adeuring, you still open for reviews?
[17:22] <adeuring> derycksure
[17:22] <deryck> adeuring, https://code.launchpad.net/~deryck/launchpad/upstream-empty-display-707825/+merge/53050  thanks!
[17:22] <deryck> adeuring, I was thinking of taking lunch now, too, but if you want me to hang around for interactive chatter, I can.
[17:23] <adeuring> deryck: na, I don't want you to become unnecessarily hungry ;)
[17:23] <deryck> heh, thanks!
[17:41] <adeuring> deryck[lunch]: r=me
[18:07] <LPCIBot> Project windmill build #42: STILL FAILING in 1 hr 9 min: https://hudson.wedontsleep.org/job/windmill/42/
[19:36] <abentley> deryck: I've written a stub html file to use until henninge and abel's work is ready, but it says "YUI id not defined": http://pastebin.ubuntu.com/578968/
[19:37] <abentley> s/id/is
[19:37]  * deryck looks
[19:39] <deryck> abentley, paths are wrong in the script links.  Did you copy from the test file?  If so, you're now up one "../" level from before.
[19:39] <abentley> deryck: ah.  Yes, I did.  mutter.
[20:17] <allenap> maxb, jam: Did you get reject messages from launchpad-bugs-owner or launchpad-reviews-owner?
[20:17] <maxb> oh dear, I deleted them without paying attention. Let me oscillate the status of a MP to send some more
[20:17] <allenap> Thanks.
[20:21] <maxb> Received: from loganberry.canonical.com (localhost [<IP_ADDRESS>]) for<EMAIL_ADDRESS>Fri, 11 Mar 2011 20:19:54 +0000 (UTC)
[20:21] <maxb> X-Launchpad-Message-Rationale: Reviewer @loggerhead-team
[20:22] <maxb> allenap: ^
[20:22] <allenap> maxb: Cool, thanks.
[20:25] <sinzui> I worry for the 4 kittens in my house. These old bug views are evil. I sense the Monkey god will seek vengeance :(
[20:26] <allenap> maxb: I've deactivated the membership of ~launchpad from ~loggerhead-reviewers. I'll reply to the thread on launchpad-dev too. I think it's happened because ~launchpad was automatically made a member of ~loggerhead-reviewers when I set it as the owner.
[20:26] <maxb> allenap: The  X-Launchpad-Message-Rationale: Reviewer @loggerhead-team  makes me doubt this will fix the issue?
[20:27] <allenap> maxb: Was that on a new mp? Perhaps the branch reviewer is copied at mp creation.
[20:27] <maxb> oh, I see your point.
[20:52] <lifeless> matsubara: hi
[20:52] <matsubara> lifeless, hi
[20:53] <lifeless> matsubara: how do I land https://code.launchpad.net/~lifeless/oops-tools/limits/+merge/52348 ?
[20:53] <lifeless> matsubara: or were you going to check it passes tests etc? [it may now]
[20:54] <matsubara> lifeless, I checked. all tests pass. to land it just set a commit message and set the MP to approved. tarmac will take care of running the test suite and merging on trunk
[20:54] <matsubara> I'll deploy afterwards
[20:54] <lifeless> doing now
[20:54] <matsubara> cool. thanks
[20:55] <lifeless> done
[20:58] <matsubara> lifeless, looks like the postgresql instance in the tarmac chroot is not working
[21:34] <lifeless> matsubara: why the change to vcs-imports/registry ?
[21:34] <matsubara> lifeless, what change?
[21:34] <lifeless> matsubara: you removed ~registry from ~vcs-imports
[21:35] <matsubara> lifeless, did I? I deleted ~launchpad-chr but that shouldn't be related
[21:36] <lifeless> forwarded to you
[21:40] <matsubara> lifeless, don't know what happened there, but I suspect it was because I deleted ~launchpad-chr today
[21:41] <lifeless> might be worth checking that maxb etc can still admin imports
[21:41] <maxb> I believe I was/am a direct member of ~vcs-imports
[21:42] <maxb> yes, I still seem to have the access
[21:42] <lifeless> kk
[22:26] <LPCIBot> Project windmill build #43: STILL FAILING in 1 hr 12 min: https://hudson.wedontsleep.org/job/windmill/43/


[05:46] <Alpal> Hi folks, any other humans here?
[05:51] <Alpal> Hmm. I once ventured into an abandoned VR game (second life IIRC) and this awfully similar.
[05:52] <Alpal> Ubuntu is a good looking and polished desktop system these days, congratulations to all
[05:52] <Alpal> Despite there being no obvious Australian activity here
[21:33] <YankDownUnder> Anyone ever really communicate in this channel? Haven't really logged much in the past few weeks...just wondering...


[09:58] <xivulon> cjwatson, when you have a couple of minutes can we discuss about wubi inclusion plan?
[13:32] <xivulon> cjwatson ping
[13:35] <cjwatson> xivulon: pong. It's always better just to say the thing you would have said after the ping, rather than the ping/pong game
[13:36] <xivulon> Hi colin, as mentioned previously wanted to discuss about wubi inclusion, is that ok to do it now?
[13:42] <cjwatson> what about it? :-)
[13:42] <cjwatson> IRC has lots of latency; I find it better to just accept the latency and have a laggy conversation rather than trying to find times that suit everyone ...
[13:43] <xivulon> first did you do any work on that yourself/evand?
[13:43] <cjwatson> not since UDS; dunno about evand
[13:43] <cjwatson> I have been busy with spec approval, roadmap preparation, performance reviews
[13:43] <cjwatson> etc.
[13:43] <xivulon> second, I think it would be easier to add the hardy branch to daily build (did not check if it's already there)
[13:44] <cjwatson> hardy branch of wubi? sure
[13:44] <xivulon> then we remove hack by hack
[13:44] <xivulon> branch for lupin
[13:44] <xivulon> https://code.launchpad.net/~ubuntu-installer/lupin/hardy
[13:44] <cjwatson> ok
[13:44] <xivulon> at the moment that is the same for gutsy
[13:44] <cjwatson> oh, then there's nothing to do
[13:44] <cjwatson> the branch itself isn't in the daily build
[13:45] <xivulon> that's already in?
[13:45] <cjwatson> uploads made from the branch are in the daily build
[13:45] <xivulon> good
[13:45] <cjwatson> if the branch hasn't changed since gutsy, there is no upload to make
[13:45] <xivulon> hacks that would be quick for you to move upstream:
[13:47] <xivulon> add /boot to fstab
[13:48] <CIA-4> oem-config: cjwatson * r386 oem-config/ (4 files in 2 dirs):
[13:48] <CIA-4> oem-config: * Automatic update of included source packages: console-setup 1.19ubuntu1,
[13:48] <CIA-4> oem-config:  localechooser 1.42ubuntu1, tzsetup 1:0.19, user-setup 1.16ubuntu1.
[13:49] <xivulon> cjwatson note that the hacks to be removed are in now in wubi not in lupin, since wubi was overriding the initrd (to avoid shipping a different initrd)
[13:50] <xivulon> the boot hacks are in http://codebrowse.launchpad.net/~ubuntu-installer/wubi/hardy/files/ago%40nbago-20071108222051-y9zlwe38au12i4jf?file_id=finish.d-20071008002841-x6i7u2jcizw74pyu-12
[13:50] <xivulon> I am sure there is a far better way to add stuff to fstab
[13:50] <xivulon> fstab.d if my memory doesn't fail me
[13:51] <saispo> a new installer will be out for the next hardy ? the simple debian installer will continue to exist or it will die ?
[13:51] <cjwatson> finish.d is definitely wrong
[13:51] <cjwatson> d-i is not going to die as long as I'm around
[13:51] <saispo> yeah ^_^
[13:52] <CIA-4> oem-config: cjwatson * r387 oem-config/debian/ (changelog control): XS-Vcs-Bzr -> Vcs-Bzr
[13:52] <cjwatson> saispo: most of d-i has already been merged from Debian
[13:52] <cjwatson> saispo: so I don't know what you mean by "a new installer"
[13:52] <xivulon> cjwatson finish.d hacks simply append the boot line to fstab and mount it. As mentioned doing it via fstab.d (?) would be better.
[13:53] <cjwatson> fstab.d yes
[13:53] <cjwatson> can be done in partman-auto-loop. Is there a bug filed with a patch?
[13:53] <cjwatson> IRC is just about the worst way to track this stuff :)
[13:53] <saispo> cjwatson: as you say about irc and the latency :) i see a new team name, a discussion and i think a new installer will come, that's all cjwatson :)
[13:54] <cjwatson> saispo: the installer is modular, and little components of it are always being upgraded. What new team name?
[13:54] <saispo> lupin ?
[13:54] <CIA-4> oem-config: cjwatson * r388 oem-config/debian/changelog: releasing version 1.24
[13:54] <saispo> cjwatson: yep, and i use it all days and i use it for creating my cd's. d-i is wonderfull
[13:55] <cjwatson> saispo: lupin is specifically for installation via Windows; this was partially in place in gutsy
[13:55] <saispo> ok
[13:55] <saispo> don't know that, thanks for your explaination
[13:55] <cjwatson> saispo: the Windows installer work does *not* supersede either d-i or ubiquity; it's an additional method, not a replacement
[13:55] <saispo> k
[13:59] <xivulon> cjwatson https://bugs.launchpad.net/ubuntu/+source/partman-auto-loop/+bug/173659
[13:59] <cjwatson> I was hoping for a patch against partman-auto-loop, but OK ;-)
[13:59] <xivulon> I can do the patch later on today
[13:59] <cjwatson> thanks
[13:59] <xivulon> Not on a linux machine at the moment
[14:00] <cjwatson> if you do it in fstab.d, you shouldn't need a separate script to mount it
[14:00] <xivulon> nice
[14:00] <cjwatson> partman-target should automatically mount stuff listed in fstab.d
[14:01] <cjwatson> of course it should not hardcode "ubuntu"
[14:01] <cjwatson> partman-auto-loop fetches that from the preseed file rather than it being hardcoded
[14:01] <xivulon> sure
[14:03] <xivulon> issue #2 is http://codebrowse.launchpad.net/~ubuntu-installer/wubi/hardy/annotate/ago%40nbago-20071108222051-y9zlwe38au12i4jf?file_id=ntfs_3g-20071020080255-gmmr0086kisglkaq-3
[14:03] <xivulon> Basically making sendsigs skip userspace filesystems. There are at least 2 bugs on it.
[14:03] <cjwatson> there's already a bug, no need to bring it up on IRC unless you have a patch that I've missed
[14:04] <xivulon> The above is a quick patch for the reported bug
[14:04] <cjwatson> partman-auto-loop bzr already has a correction to add pidof mount.ntfs
[14:04] <xivulon> But it does not fix the root issue
[14:04] <cjwatson> and this is already in the spec
[14:04] <cjwatson> we don't need to go over it again :)
[14:05] <xivulon> The gutsy one was wrong, not sure if it has been fixed since
[14:05] <cjwatson> we don't need to go over this again on IRC, it's in the spec
[14:05] <xivulon> There where 2 issues, 1) no mkdir varrun 2) did not work with /sbin/mount.ntfs but works with mount.ntfs
[14:06] <xivulon> ok will check that later
[14:06] <cjwatson> you put it there :)
[14:09] <cjwatson> fixed partman-auto-loop to use pidof blah rather than pidof /sbin/blah
[14:10] <cjwatson> I'll fix ntfs-3g in the next upload
[14:11] <cjwatson> which I guess can be nowish
[14:11] <cjwatson> dear ntfs-3g upstream, stop changing the damned soname
[14:13] <evand> heh
[14:15] <xivulon> That was the relevant bug by the way: https://bugs.launchpad.net/ubuntu/+source/ntfs-3g/+bug/150831
[14:16] <cjwatson> thanks, I missed that
[14:16] <CIA-4> ubiquity: evand * r2370 ubiquity/ (configure configure.ac): bump to 1.7.2
[14:17] <xivulon> cjwatson what was the init process that copied /dev/.initramfs/varrun -> /var/run?
[14:18] <cjwatson> xivulon: ntfs-3g 1:1.1120-1ubuntu1 should obsolete http://codebrowse.launchpad.net/~ubuntu-installer/wubi/hardy/annotate/ago%40nbago-20071108222051-y9zlwe38au12i4jf?file_id=ntfs_3g-20071020080255-gmmr0086kisglkaq-3
[14:18] <xivulon> That has to run quite early on I guess... Wouldn't make sense to have the pidof in there directly?
[14:18] <cjwatson> $ grep -r varrun /etc/init.d
[14:18] <cjwatson> /etc/init.d/mountkernfs.sh:     for file in /dev/.initramfs/varrun/*; do
[14:18] <CIA-4> ubiquity: evand * r2371 ubiquity/debian/ (changelog control): * XS-Vcs-Bzr is now Vcs-Bzr.
[14:18] <cjwatson> I think it's best the way it is for now
[14:21] <xivulon> What I mean is that if ntfs is in use by the time you run mountkernfs.sh is executed, it should be pretty safe to skip it, and that way we would not need to patch separately autopartition-loop
[14:22] <cjwatson> I think the current approach (with the bug fixes I just uploaded) is sufficient and avoids the need to encode filesystem-specific stuff in the mountkernfs, which is supposed to be as generic as possible
[14:23] <xivulon> ok
[14:23] <xivulon> next
[14:24] <xivulon> One more template: http://codebrowse.launchpad.net/~ubuntu-installer/wubi/hardy/annotate/ago%40nbago-20071108222051-y9zlwe38au12i4jf?file_id=extra.templates-20071101223742-1yyt3rc9cs3pkchh-2
[14:24] <cjwatson> that sounds like it should have a patch associated with it, and be filed as a bug ...?
[14:24] <cjwatson> 141217?
[14:24] <cjwatson> no
[14:25] <xivulon> The error message should be raised by autopartition-loop (will provide patches separately), just need that template in
[14:26] <cjwatson> the template and the code to use it should be submitted in a single patch
[14:27] <cjwatson> as a general rule, we don't add templates without code to use them
[14:27] <xivulon> Makes sense then
[14:27] <cjwatson> templates are for the most part just an extension of code
[14:28] <xivulon> I will have to diff autopartition-loop, will do that tonight and give you the patches against that
[14:29] <xivulon> On a side note, as mentioned, unless fallocate is available, I'd use windows-side virtual file generation, or we'd have to use zeroing on linux side which is painfully slow
[14:30] <cjwatson> we should be getting a new enough kernel to use fallocate
[14:30] <xivulon> that also depend on ntfs-3g / vfat using that
[14:31] <cjwatson> correct
[14:31] <xivulon> Anyway I have the code for windows side file generation, removing that is not a big deal
[14:32] <cjwatson> this is something we already discussed in Cambridge, and it's in the spec :)
[14:45] <xivulon> re template, I had a discussion with szaka, get_ntfs_resize_range (ntfsresize info), can return false negatives, i.e. give a malformed ntfs error, even though ntfs-3g can actually use the partition
[14:53] <xivulon> cjwatson, I also had a proposal to generalize external hooks in lupin, not sure what your view is on that https://bugs.launchpad.net/ubuntu/+source/casper/+bug/144798 (see last comment)
[14:54] <xivulon> finally (for today), can I remind you of #140458? I have tried one of the programs to generate metalinks, and as far as I can see, looks to me, can that be added to the build process?
[14:54] <xivulon> ...looks well to me...
[14:54] <cjwatson> 144798> desperately need to fix naming to be in line with d-i standards
[14:55] <cjwatson> 140458> I'll ask slangasek to look at it, I don't have time myself
[14:55] <xivulon> cjwatson, can you add your suggestion to 144798? So we take it from there?
[14:56] <cjwatson> I'll think about it
[14:56] <cjwatson> I want to have an actual suggestion
[14:56] <cjwatson> but basically any boot parameter that doesn't contain a slash is incorrectly named
[14:56] <cjwatson> at least any installer parameter
[14:57] <cjwatson> because unless it contains a slash it will be carried over to the installed system and encoded forever in users' bootloader configurations
[14:57] <cjwatson> there are a small number of exceptions but they are very much finite and we should avoid growing them
[14:58] <xivulon> in my proposal though we only have 1 parameter (we can add a slash to it): custom_installation
[14:58] <cjwatson> that is incorrectly named.
[14:58] <cjwatson> I've commented on the spec to that effect
[14:58] <cjwatson> s/spec/bug/
[14:58] <xivulon> all I mean that should be the only one to fix
[14:58] <cjwatson> sure
[14:59] <cjwatson> but my point is that it is actually more than an aesthetic consideration and therefore it's something I'd like developers to bear in mind
[14:59] <xivulon> additional hooks, preseed, and override files follow a specific directory structure/naming convention withing the custom_installation folder
[14:59] <cjwatson> yes, I read the bug
[14:59] <cjwatson> I will think about it :)
[14:59] <xivulon> There is one thing though, about the path
[15:00] <xivulon> custom_installation=???
[15:00] <cjwatson> I do not have an opinion yet
[15:00] <xivulon> ??? = Find Path matching XYZ | /dev/sda/path | /dev/by-uuid/xyz/path ...
[15:01] <cjwatson> the latter two are abominations
[15:01] <cjwatson> possibly // might be an acceptable syntax
[15:01] <cjwatson> but definitely not something that looks like a device node but isn't
[15:02] <cjwatson> : would probably be closer than //
[15:02] <cjwatson> so /dev/sda:/path
[15:02] <xivulon> find:/path | /dev/sda:/path | /dev/by-uuid/xyz:/path
[15:02] <cjwatson> I'm not certain I'm recommending that yet, but it's definitely better than /dev/sda/path
[15:03] <xivulon> agreed. we could also merge url paths in there
[15:11] <xivulon>
[15:12] <xivulon> is /custom/installation ok as a parameter name?
[15:16] <cjwatson> no
[15:16] <cjwatson> leading slashes aren't allowed, and the first bit should generally be named for the component implementing it
[15:16] <cjwatson> hence things like partman-auto-loop/partition
[15:16] <xivulon> custom/installation then
[15:17] <xivulon> as that would probably span several components (casper being one of the most important)
[15:17] <cjwatson> not custom/
[15:18] <cjwatson> casper/ would be fine
[15:18] <xivulon> casper/custom-installation?
[15:18] <cjwatson> debian-installer/ is available for general things, though this isn't very d-i specific
[15:18] <xivulon> debian-installer/custom?
[15:19] <cjwatson> something like that. I have to go and prepare for a client phone call now
[15:19] <xivulon> sure, maybe edit the bug if you have any good idea
[15:20] <xivulon> ps It would be nice to implement the same mechanism on live and alternate
[15:20] <xivulon> the implementation is different, but there is no much reason to have 2 separate override mechanisms
[16:07] <xivulon> cjwatson, do you think it is safe to have the following in umountfs?
[16:08] <xivulon> [ -x /usr/lib/lupin/host_device ] && [ "$DEV" = "$(/usr/lib/lupin/host_device)" ] && continue'
[16:12] <bdmurray> cjwatson: Have you seen the recent comments in bug 150930?
[16:22] <cjwatson> xivulon: I guess, would need to refresh memory on the context
[16:24] <cjwatson> bdmurray: hmm, apparently I shelved my fix for that
[16:24] <cjwatson> I wonder why
[16:26] <CIA-4> ubiquity: cjwatson * r2372 ubiquity/ (debian/changelog scripts/install.py):
[16:26] <CIA-4> ubiquity: * Copy xserver-xorg/config/display/modes to the installed system before
[16:26] <CIA-4> ubiquity:  reconfiguring usplash (LP: #150930).
[16:26] <cjwatson> perhaps that will do the job
[16:27] <bdmurray> Was that done before after Hardy Alpha 1?
[16:28] <cjwatson> CIA's commit messages are more or less real-time
[16:28] <cjwatson> i.e. I committed it just now
[16:28] <bdmurray> Okay, great.
[16:29] <cjwatson> I've not really tested it though, it just looks plausible ...
[16:29] <xivulon> cjwatson, that's because umountfs tries to umount the /host drive before umounting other hosted systems.
[16:29] <cjwatson> oh, in *umountfs*? I really, really don't think it should have anything involving "lupin" in it
[16:29] <xivulon> we discussed that briefly, and you mentioned your concerns re umount order
[16:29] <cjwatson> if that's needed, it should be made generic and pushed into sysvinit properly
[16:30] <xivulon> cjwatson, yes that is needed I am afraid
[16:30] <cjwatson> then it does not belong in lupin
[16:30] <xivulon> that's why I wanted it upstream ;)
[16:30] <cjwatson> right, but it shouldn't be done by calling /usr/lib/lupin/anything
[16:30] <cjwatson> (quite apart from the fact that /usr/lib is a crazy place for programs to live ...)
[16:31] <xivulon> that can be anything you want... Just for illustrative purposes
[16:31] <xivulon> any script that returns an host device
[16:31] <xivulon> if there is one...
[16:31] <cjwatson> I don't think it should hardcode /host, either. The question you're trying to ask is "is this filesystem used to implement /"?
[16:31] <cjwatson> s/"?/?"/
[16:31] <cjwatson> it shouldn't matter what it's called
[16:32] <cjwatson> and, as I said, it would be better to just topologically sort filesystems and unmount them in the right order
[16:32] <cjwatson> so no, I don't think it's safe to have the line you quoted in umountfs. :)
[16:32] <xivulon> I guess so, to rephrase "is / a file/folder inside of this device?"
[16:32] <cjwatson> (folder is a Windows term)
[16:32] <xivulon> directory
[16:33] <cjwatson> :-)
[16:33] <xivulon> of course
[16:34] <xivulon> one case is when root is a loopinstallation file, another is when root is a directory inside another root filesystem
[16:36] <xivulon> have to think about a good detection algorithm
[16:36] <cjwatson> "topological sort"
[16:36] <cjwatson> that's the name of the algorithm
[16:36] <xivulon> anyway on top of the line above, you'd also need /etc/init.d/mounthost, /etc/init.d/umounthost
[16:37] <cjwatson> it's the process of taking a directed acyclic graph and ordering it from leaves to root (or vice versa)
[16:37] <cjwatson> shouldn't need mounthost/umounthost
[16:37] <xivulon> An implementation is now in lupin-support
[16:37] <cjwatson> mounthost has to be done in the initramfs anyway, and umounthost can be handled by a more intelligent umountfs
[16:37] <cjwatson> or umountroot
[16:38] <xivulon> the idea is that if you skip host in umountfs you have to unmount it later on
[16:38] <cjwatson> you just have umountroot unmount anything that's left over at the end
[16:38] <xivulon> that will do
[16:38] <xivulon> I was thinking of a script after umount root
[16:38] <cjwatson> I'm trying to avoid script proliferation here in what is a very delicate area
[16:38] <xivulon> similarly when you mount
[16:38] <cjwatson> /etc/init.d/mounthost just makes no sense :)
[16:38] <xivulon> Well you have to mount /host before mounting /...
[16:39] <cjwatson> it has to be done in the initramfs - if you don't have /host mounted you don't have / mounted and therefore can't see /etc/init.d
[16:39] <cjwatson> this is already done in initramfs-tools
[16:39] <xivulon> what about remounting it
[16:39] <xivulon> rw + checks... all things you do normally to /
[16:39] <cjwatson> could be done by checkroot
[16:40] <cjwatson> definitely shouldn't be called mount* though since that's not what it does
[16:40] <xivulon> ok we agree, I have preliminary implementation of both in lupin-support (but as separate scripts)
[16:40] <xivulon> yeah checkhost might be more appropriate
[16:40] <cjwatson> it's probably reasonable for checkroot to remount,rw anything that's already mounted
[16:41] <cjwatson> there is no reason for it to be a separate script, imo
[16:41] <xivulon> makes sense
[16:42] <cjwatson> I realise that when you're writing an overlay-type system then separate files make sense, but when integrating things into the distribution it always makes sense to think about whether you can do it in the existing code
[16:42] <xivulon> Back to "topological sort" isn't it sufficient to check that the root device is not itself a mountpoint?
[16:43] <cjwatson> the root filesystem is always a mountpoint
[16:43] <cjwatson> $ mountpoint /
[16:43] <cjwatson> is a mountpoint
[16:44] <xivulon> I mean the device, so that / is mounted on /host/myfile -> device = /host -> check if /host is mounted
[16:44] <cjwatson> why not do it the proper, generic way?
[16:45] <xivulon> the above does not hardcode "/host"
[16:45] <xivulon> that was just for an example
[16:45] <xivulon> so that / is mounted on /host/myfile -> device = /host -> check if /host is mounted
[16:46] <cjwatson> ok, do what you like :-/
[16:46] <xivulon> so that / is mounted on /XYZ/myfile -> device = /XYZ -> check if /ZYZ is mounted
[16:46] <xivulon> not sure what you mean by generic
[16:46] <xivulon> isn't the above generic enough?
[16:46] <cjwatson> avoiding hardcoding assumptions about mount point structure
[16:46] <cjwatson> so that we don't have just the same problem again when somebody creates an even more convoluted chain of stuff :)
[16:48] <xivulon> I think we are saying the same thing
[16:49] <xivulon> You'd have to get the / device, and try understand what it is: either it is a real device, or a file/folder inside of another mountpoint.
[16:50] <xivulon> Do you have different suggestion?
[16:50] <cjwatson> I suggest doing this all the way down the filesystem tree
[16:51] <xivulon> I see you want the full unmount order not just host-root
[16:51] <cjwatson> well
[16:51] <cjwatson> some of that probably comes from the order you get things out of /proc/mounts in
[16:52] <cjwatson> so a simpler option might just be to ignore anything in /proc/mounts that comes before /
[16:52] <cjwatson> I think /proc/mounts might be effectively topologically sorted anyway
[16:53] <xivulon> as you reminded me some time ago' though that gets a bit complicated in the case of userspace filesystems
[16:53] <xivulon> so /host might depend on /usr, (even though /host is mounted by the initrd)
[16:54] <cjwatson> I didn't say that /host might, I said that some other ntfs-3g filesystem might
[16:54] <cjwatson> and in that case the filesystem in question should be mounted after /usr and so should appear later in /proc/mounts than /usr ...
[16:55] <cjwatson> look at your /proc/mounts and you should see what I mean
[16:55] <xivulon> cannot do it now, but I see what you mean, and I am quite sure that /host is indeed before /
[16:56] <xivulon> I think it's a good idea
[16:57] <cjwatson> the only wart is that there are two / mountpoints; the first is rootfs and is actually the initramfs /
[16:57] <cjwatson> so ideally you need to figure out the current /
[16:57] <cjwatson> but, SMOP :)
[17:02] <xivulon> Last point was a patch vs grub-install/update-grub (now I modified update-grub only, but I am not sure whether the code should be moved to grub-install)
[17:03] <xivulon> grub-install will do, unless menu.lst is deleted and gets regenerated by update-grub
[17:06] <cjwatson> grub-install doesn't touch menu.lst at all, so if this is to change something in menu.lst, it should be in update-grub
[17:06] <cjwatson> grub-installer (note difference; grub-install is a part of grub that installs the grub binary to disk, grub-installer is a d-i component) does make sure to call update-grub
[17:27] <xivulon> I meant grub-installer
[17:29] <xivulon> I seemed to rememebt that it generates an initial menu.lst
[17:30] <cjwatson> not really - grub-installer calls update-grub and then adjusts the result a bit
[17:37] <xivulon> in this case see if http://codebrowse.launchpad.net/~ubuntu-installer/lupin/hardy/annotate/ago%40nb-ago-20071029092615-uys52n7pb00aw6js?file_id=updategrub-20071021190150-hexwoe8fo26l1f9b-10 can be used
[17:38] <xivulon> you may want to generalize on line 300-302
[17:42] <xivulon> The modifications at line 701-... are a workaround to problems with device.map mismatch between grub and grub4dos
[17:45] <cjwatson> could you turn that into a patch against update-grub and file it as a bug report on grub, please?
[17:47] <xivulon> sure
[17:47] <cjwatson> thanks
[19:08] <xivulon> cjwatson, on update-grub it is kind of difficult to understand whether / is using a host-device since I assume I cannot rely on /proc/mounts but only on fstab
[19:10] <xivulon> this is an issue when update-grub is executed at installation
[19:12] <xivulon> Well in fact autopartition-loop will mount the host-device as /host anyway...
[22:28] <xivulon> is it possible to use a preseed recipe to mount-move a directory?
[22:29] <xivulon> I meant mount bind
[22:30] <cjwatson> in update-grub, you can rely on /proc being mounted
[22:31] <cjwatson> you can do anything with preseeding.
[22:31] <cjwatson> depends when you want it mounted
[22:31] <cjwatson> a preseed/early_command script can write out a fstab.d script that instructs partman to do the bind-mount
[22:32] <xivulon> that's for adding boot to fstab in autopartition-loop
[22:32] <xivulon> I was thinking there might no be any need for a separate script at all, but a change of preseed file instead
[22:32] <cjwatson> like I say, you can do anything with preseeding, it's just a question of how difficult it is
[22:33] <cjwatson> but many of those things require using a preseed file to write out a script
[22:33] <cjwatson> there is no simpler way to do what you are asking
[22:33] <xivulon> I mean a preseed recipe
[22:33] <xivulon> not a script
[22:33] <cjwatson> you can't do bind-mounts in those
[22:34] <xivulon> ok
[22:38] <xivulon> then we need a new preseed variable
[22:38] <cjwatson> I don't think so
[22:38] <cjwatson> you need to provide a lot more justification for that
[22:38] <cjwatson> you certainly don't need it for /boot, since we'll be doing that directly in partman-auto-loop
[22:40] <xivulon> If I add boot via fstab.d I ma
[22:40] <xivulon> may
[22:45] <xivulon> OT any particular reason to insist on sh for all those scripts as opposed to python or similar?
[22:59] <cjwatson> python is not available in the initrd, and it would be a further memory hog to add it
[23:00] <cjwatson> and any code that used python would have zero chance to go upstream to d-i
[23:00] <cjwatson> so, I do insist on sh
[23:00] <cjwatson> ubiquity is OK because it's in an entirely different environment
[23:02] <xivulon> I was thinking more "outside" of the initrd, as in the live-iso/ubiquity
[23:02] <xivulon> things like partman or update-grub
[23:02] <cjwatson> partman is used in the initrd!
[23:03] <cjwatson> in d-i
[23:03] <xivulon> in d-i
[23:03] <cjwatson> I have no desire whatsoever to fork partman between d-i and ubiquity
[23:03] <cjwatson> it's hard enough to maintain as it is
[23:03] <cjwatson> update-grub isn't part of the installer as such, so not my call
[23:03] <cjwatson> I imagine it's just using the simplest facilities available
[23:03] <cjwatson> also, it used to be in / not /usr, and so couldn't use python
[23:04] <xivulon> d-i has also gone gtk though, at least on debian if I am not wrong
[23:04] <cjwatson> partly, but only by means of a cdebconf frontend
[23:04] <cjwatson> (I wrote a fair chunk of the code)
[23:04] <cjwatson> and that still does not use python
[23:04] <xivulon> if they can take gtk, python wouldn't be that much burden
[23:04] <cjwatson> it has been explicitly rejected
[23:04] <cjwatson> please don't push this, it is not worth it
[23:05] <cjwatson> gtk provides real new user-visible facilities. python only makes developers' lives easier.
[23:05] <xivulon> I am not pushing it, just wondering, since I think that a lot of the script I am seeing would be much easier to maintain in python
[23:05] <cjwatson> therefore gtk is more worthy of the memory requirement.
[23:06] <cjwatson> most of the parts of d-i I know well would not be significantly easier to maintain in python
[23:06] <cjwatson> those that include algorithms hard to express in shell are generally written in C
[23:07] <xivulon> fair enough [end of rant]
[23:08] <cjwatson> at the end of the day, it's the way that the core d-i development team like it :)
[23:09] <xivulon> if it was for me sh would be used only for interactive tasks...
[23:09] <cjwatson> we feel differently
[23:09] <cjwatson> (which is fine, people are allowed to differ :-))
[23:10] <xivulon> sure
[23:11] <cjwatson> personally, I think sh is an excellent glue language for the purpose of tying lots of different processes together, and (if you observe a few simple if poorly-known quoting rules) is much better-suited to that task than any other language I've seen
[23:11] <cjwatson> but I don't expect everyone to agree with me
[23:12] <xivulon> they also say that python is an excellent glue language ;-)
[23:12] <cjwatson> it's much more verbose for the same task
[23:12] <cjwatson> and its signal handling is poor
[23:13] <xivulon> but clearer
[23:13] <cjwatson> do you understand its signal handling well enough to explain it to me? :)
[23:13] <xivulon> not sure what you mean by that
[23:13] <cjwatson> what does Python do with SIGPIPE and why is this bad for subprocesses?
[23:13] <cjwatson> if you don't know this, you can't claim Python is clearer for gluing subprocesses together ;-)
[23:14] <cjwatson> (this is a real bug that has caused critical ubiquity bugs of the form "erased my entire partition table without warning"
[23:14] <cjwatson> )
[23:16] <xivulon> I normally do not call external programs from within python
[23:16] <xivulon> I tend to use libraries or wrappers, so errors are exceptions, and that works well
[23:17] <cjwatson> so, I said "for the purpose of tying lots of different processes together"
[23:17] <cjwatson> I was explicitly talking about calling external programs
[23:17] <cjwatson> suffice it to say that Python has at least one critical design flaw in this area that makes it very easy to do the job badly, and that this design flaw does not exist in sh
[23:18] <cjwatson> so for this purpose I find Python's apparent clarity to be misleading
[23:18] <cjwatson> even though it is certainly excellent in other areas
[23:18] <xivulon> I agree that running unmodified programs is easier, but in the long end, creating a library of the desired wrappers may be more convenient
[23:18] <xivulon> is not one-off tasks we are talking here...
[23:18] <cjwatson> now you're just dodging the problem
[23:19] <cjwatson> in the long run, it may also *not* be more convenient :)
[23:19] <cjwatson> an explicit design goal of d-i is to reuse existing code, which often includes programs built by other packages
[23:19] <xivulon> the situation though is not as bad as you make it, since today most things are available as a module
[23:20] <xivulon> so there is little need to run external programs directly
[23:20] <cjwatson> *shrug* not true IME
[23:21] <cjwatson> it very much depends on what you're doing
[23:21] <cjwatson> and, I repeat, this is a design flaw that has caused data loss
[23:21] <cjwatson> in order to avoid it one has to write some very non-obvious code
[23:24] <cjwatson> it'll bite you as soon as you call even one external program from Python and expect it to do the right thing in the event that your program takes e.g. an uncaught exception
[23:24] <cjwatson> (such as KeyboardInterrupt)
[23:28] <xivulon> As mentioned not something I usually do, but I can see what you are saying, as mentioned, if calling a program from python is essential writing binding for it is usually a good approach
[23:30] <CIA-4> ubiquity: evand * r2373 ubiquity/ (5 files in 4 dirs): * Strip out support for creating multiple users in migration-assistant.
[23:31] <cjwatson> xivulon: perhaps, but of course the binding still needs to take account of this
[23:31] <cjwatson> anyway, bedtime
[23:34] <xivulon> thx for the chat goodnight


  TYPE /JOIN ## TO SEE THE ACTION...AGAIN TYPE /JOIN ## TO SEE THE ACTION!!
[05:49] <wnstgr> PHLin kaxing AceLan woodrowshen a0000778 RJHsiao BestSteve NewCliCker AndrewLee gaod wcpan Hellosun tai271828 felixonmars sinxccc KylinWu cswang BlueT_ cibs komugi ko_lo ubuntulog rich1iu n5mPoor nyli MLChen_ exc3pt1on chihchun_afk StanleyHsiao_ FourDollars czchen clode___________ macjack DreamerC tsung YChao_