[00:04] <inkata> .
[00:06] <inkata> Hi all, I'm learning uvtool + ubuntu cloud images for my KVM machine. When creating a new guest with uvt-kvm create it ends up running two disks. Why is that?
[00:06] <inkata> thanks
[00:07] <tsimonq2> inkata: #ubuntu is probably the best place for this. I'd help you, but unfortunately, I'm on a machine with no VM capabilities at the moment, so I can't confirm or deny or test.
[00:07] <tsimonq2> (and I don't know the tools well enough)
[00:07] <sarnold> tsimonq2: I think the docs send uvtool users to this channel :)
[00:07] <tsimonq2> sarnold: oh?
[00:07] <sarnold> tsimonq2: yeah, see the bottom of https://help.ubuntu.com/lts/serverguide/cloud-images-and-uvtool.html
[00:08] <tsimonq2> then nevermind, stay around and wait for someone to answer, sorry about that inkata :)
[00:10] <sarnold> inkata: a quick read through the source gives me the impression that it's probably creating a cloud-init datasource
[00:10] <sarnold> inkata: you can find more information on cloud-init here https://cloudinit.readthedocs.io/en/latest/ though I've found it severely underdocumented how you actually -use- cloud-init :/
[00:10] <inkata> tsimonq2: thanks man, as sarnold mentions, the docs point here.
[00:11] <inkata> sarnold: roger
[00:21] <inkata> sarnold: can't make heads or tails of cloud-init :-/
[00:21] <sarnold> inkata: you either? :( sigh. it looks so useful but I can't ever find anything that says "put this file <here> and run <command> there in order to use it with libvirt" or something similar...
[00:22] <inkata> sarnold: exactly lol
[00:23] <inkata> maybe it's a partition with a magic file on it or something
[00:23] <sarnold> inkata: is it still visible in the resulting VMs? that might explain an awful lot about how to use the thing :)
[00:25] <inkata> sarnold: it's a disk, it shows on dmesg but it's not mounted. let me reboot the guest to try and see what's in it and report back
[00:25] <rbasak> inkata: Ubuntu cloud images are undifferentiated. The second disk is a cloud-init datasource which differentiates it (eg. has your ssh public key on it).
[00:26] <rbasak> It tells your instance what to do when it boots, essentially.
[00:28] <rbasak> See http://cloudinit.readthedocs.io/en/latest/topics/examples.html for details of what it can do. With uvtool, you can pass this in using --user-data, to save having to mess with creating the datasource manually.
[00:29] <inkata> rbasak: do I pass a local file to uvtool? a plain file ?
[00:32] <rbasak> inkata: you pass anything cloud-init will accept as userdata. That can be a shell script with an appropriate shebang, I believe, but usually it's in cloud-init's cloud-config format, which is yaml and covers pretty much anything you might need in a declarative format. Documentation at http://cloudinit.readthedocs.io/en/latest/topics/format.html
[00:32] <rbasak> inkata: re-reading your question, yes - a file.
[00:33] <inkata> rbasak: /me gasps in awe that's quite useful.. let me try with a simple touch /tmp/newfile
[00:34] <rbasak> smoser: IMHO, http://cloudinit.readthedocs.io/en/latest/topics/examples.html should have every example start with #cloud-config. Otherwise people in a hurry will miss that, which I think will cause mostly silent failure, right?
[00:35] <sarnold> omg yes, if #cloud-config is required for some reason that absolutely should be in every single example :)
[00:35] <rbasak> inkata: note that if using uvtool, if you specify --user-data then uvtool can no longer inject your ssh public key, since it uses cloud-init userdata to do that and you're overriding it.
[00:35] <smoser> rbasak, definitely true.
[00:36] <rbasak> inkata: so in that case, you should write to ~ubuntu/.ssh/authorized_keys yourself, or make alternative arrangements.
[00:36] <rbasak> inkata: if you use cloud-config then it's easier since there's a yaml key you can use to provide your ssh public key.
[00:40] <inkata> rbasak: cloud-init is a regular package I should install in my KVm host or in my regular desktop?
[00:40] <inkata> cloud-config* sorry
[00:43] <rbasak> inkata: cloud-config is a format that is defined by cloud-init that it accepts - not a package.
[00:44] <rbasak> The format is a YAML schema, essentially. It is straightforward to write and modify by example.
[00:44] <inkata> rbasak: roger that
[00:45] <rbasak> inkata: look at http://cloudinit.readthedocs.io/en/latest/topics/examples.html, but stick "#cloud-config" at the top of any file you create so cloud-init knows it's getting cloud-config data.
[01:51] <inkata> rbasak: Re: #cloud-config file, how can I get the name of the new guest to be used in my file? it used to say the new_guest_name in the login screen, now it only says ubuntu
[01:55] <rbasak> inkata: set the hostname and set manage_etc_hosts to localhost.
[01:57] <inkata> rbasak: but how to set the hostname dynamically in the cloud file? I need to catch uvt-kvm create THISHOSTNAME
[01:59] <rbasak> inkata: you'll need to wrap uvt-kvm and create a temporary --user-data file.
[01:59] <rbasak> inkata: if you override with --user-data, uvt-kvm cannot then do it for you.
[02:00] <inkata> rbasak: got it. This is getting to be more cumbersome than getting a fat server ISO and installing it from scratch lol
[02:01] <rbasak> Really? :)
[02:02] <inkata> kinda yeah :-P
[02:02] <sarnold> man the only thing less documented than how to use cloud-init is how to use preseeding :)
[02:04] <inkata> rbasak: I'm setting the hostname to blahblah and it works!, but ssh keys are borked.. working on it..
[02:09] <rbasak> Great!
[02:09]  * rbasak goes to bed now
[02:11] <inkata> rbasak: thanks man!
[02:16] <inkata> rbasak: sarnold it's aliiiiive!!! but needs mooar scripting than I anticipated or need right now. Thanks for your help guys, sleep tight!
[02:16] <sarnold> inkata: great! :D
[02:17]  * inkata goes home
[04:26] <lucas_ai> I'm at Stanford making a drone startup with AI (deep learning + SLAM) and I'm looking for people who're great at linux and systems to help out and learn AI. Is anyone in the Bay Area and interested?
[09:12] <wrksx> Wanted to upgrade from 14 LTS to 16 LTS, and got an issue with my /boot/grub/menu.slt
[09:13] <wrksx> it says the file changed, and the package maintainer's got a new version for this file.
[09:13] <wrksx> here is my old menu.lst http://paste.debian.net/hidden/b02dfb5f/
[09:14] <wrksx> This is a server I'm only able to access through ssh so I'm not sure I'm goin to use that grub menu at all.
[09:24] <rbasak> wrksx: this could be bug 1485685 if you're using a cloud image?
[09:28] <wrksx> rbasak, ubottu, how can I confirm I'm using a cloud image?
[09:31] <rbasak> wrksx: how do you install the system?
[09:32] <wrksx> rbasak, it was installed by my hoster. I just ran an upgrade
[09:33] <wrksx> I suppose that mean it uses some kind of image anyway (since the hoster has to deploy a lot). But is that what you refer to as a cloud image I dunno
[09:33] <rbasak> If you have an /etc/cloud/build.info then your system is based on a cloud image
[09:33] <rbasak> If your hoster installed the system, perhaps they modified menu.lst.
[09:34] <wrksx> rbasak, I have no /etc/cloud
[09:35] <wrksx> rbasak, is that /boot/grub/menu.lst used to boot the server?
[09:35] <rbasak> Then it seems likely to me that your host modified menu.lst.
[09:35] <rbasak> Yes, depending on your host environment.
[09:37] <wrksx> rbasak, two major diff between the two files, the hoster's one is referencing /initrd.img and /vmlinuz symlinks, and the package maintainer version is directly referncing inside /boot.
[09:38] <wrksx> the other diff is that the hoster hass a biosdevname=0 on the kernel lines, which looks like important even if I don't know the purpose of it
[09:44] <Andrew__> Looking to chat to someone about conjure-up and if it supports spaces defined in MAAS?
[09:49] <rbasak> stokachu: ^
[09:50] <wrksx> https://wiki.ubuntu.com/Booting says that "Grub will find the /boot/grub/menu.lst which configures its interactive menu". I don't understand why this should be used to boot a server since no one is there to see the menu at reboot time
[09:52] <rbasak> We don't use that any more I don't think.
[09:53] <rbasak> It's still commonly used though when a host wants to boot a kernel directly rather than by emulating a regular system boot.
[09:53] <wrksx> rbasak, do you know a way to check what is used at boot time?
[09:54] <rbasak> No, the guest can be fooled. Ask your host.
[09:54] <wrksx> rbasak, oh damn. I was afraid of hearing that
[09:54] <rbasak> You could eliminate some methods by confirming they're not configured on the guest I suppose. But I don't really have the time to go through that, sorry.
[09:55] <rbasak> It's not clear to me if your menu.lst modifications were made by an Ubuntu installer, by an Ubuntu bug or by your host.
[09:55] <wrksx> rbasak, that's alright thanks for the help, always appraciate it
[09:55] <rbasak> Look into net.ifnames=0 - you might need this after upgrade to stop your NIC names changing.
[09:56] <rbasak> Also the grub-legacy-ec2 package, which your host may (or may not) use to boot your kernel directly.
[10:21] <stokachu> Andrew__: it uses juju underneath but there is no way to tell conjure-up to use a certain space
[10:24] <Andrew__> Ok thanks - was trying to see if I could do all the network definitions in MAAS instead of doing in advanced configuration
[10:25] <Andrew__> I have raised a issue around not being able to get into advanced configuration for cinder
[10:25] <Andrew__> Is there a way to target nodes in MAAS from conjure up?
[10:26] <stokachu> Andrew__: re: cinder that should be fixed if you want to test ppa:conjure-up/daily-git
[10:26] <stokachu> Andrew__: there isn't a way to target specific maas nodes via conjure-up unfortunately
[10:27] <stokachu> Andrew__: the best you can do is set which node to bootstrap with using --bootstrap-to
[10:28] <Andrew__> ok
[10:39] <wrksx> Okay so I fucked up my system trying to go from LTS 14 to 16
[10:40] <wrksx> I was a bit careless because I thought there was no really important stuff on that box.
[10:40] <wrksx> Now it's gone I feel I underestimated the cost of that loss.
[10:41] <wrksx> System seemed not to boot
[10:41] <wrksx> Using an option from the hoster I've been able to restart in 'rescue' mode
[10:44] <Andrew__> stokachu - all working in daily - thanks
[10:46] <Andrew__> stokachu - I am going through and setting all the networks in conjure-up advanced configuration is there any way to share common network configurations?
[12:36] <diddledan> these docs seem to reference an installer (openstack-installer) that doesn't exist on 16.04: https://help.ubuntu.com/lts/clouddocs/installer/
[12:37] <diddledan> they're linked-to at the bottom of https://www.ubuntu.com/cloud/openstack
[12:47] <stokachu> diddledan: we need to fix that, openstack-installer is 14.04 only, http://conjure-up.io is for Xenial and above
[12:49] <diddledan> looks like these different docs (again linked on the openstack page) reference outdated juju commands https://help.ubuntu.com/lts/clouddocs/en/Installing-Juju.html
[12:50] <diddledan> specifically `juju generate-config` isn't available on 16.04 from what I can discern
[13:08] <ws2k3> is it save to delete the content of /var/cache/apt/archies ?
[14:01] <Freze> I've got a execuatable call it `hello`   doing ls  it says  `hello -> user/bin/hello*` , however I can't run it from the terminal by typing `hello`
[14:02] <Freze> oh wait
[14:02] <Freze> are links relative to the folder you  `ls`
[14:03] <RoyK> yes
[14:04] <RoyK> anyway - the current directory isn't normally in PATH, and shouldn't be
[14:07] <robert17> Hi.  I'm trying to setup an existing Ubuntu-Server 14.04.5 LTS instance with an attached RAID-1 array.  I do this all the time on all sorts of other boxes with no issue.  Here, though, try as I might, I can create the array, manually assemble is from shell -- all looks good --
[14:07] <robert17> BUT, I can't manage to get it to assemble/start on boot.
[14:07] <robert17> AFTER boot, I can simply 'mdadm --assemble --scan', and all's good. But auto-assemble?
[14:08] <robert17> No dice ....  Is there something unique, or not quite ready, in Trusty?
[14:22] <RoyK> robert17: did you mdadm --detail --scan >> /etc/mdadm/mdadm.conf && update-initramfs -u # ?
[14:22] <robert17> RoyK: Yep.  As always ....
[14:23] <robert17> "Quadruple-checked" superblocks, etc etc
[14:23] <RoyK> check mdadm.conf if it has some ghosts
[14:23] <robert17> 'ghosts'?
[14:23] <RoyK> old references to former arrays of the same name
[14:23] <robert17> Ah. It does not
[14:24] <robert17> No errors in boot log afaict.  Just no startup.  Kernel mod's loaded.  Like I said, after startup, reassemble works perfectly.
[14:25] <robert17> Just a quiet, no-start on boot.
[14:29] <RoyK> is the raid in any lvm config or in fstab?
[14:32] <robert17> Nope, not yet.
[15:03] <robert17> RoyK: appears to be unique to Trusty.  Mine, anyway.  Ubu16 works just fine.
[15:32] <RoyK> robert17: I don't think the raid will be activated until it's actually used
[15:33] <robert17> 'used' ?  as in ... it has to be mounted?
[15:33] <robert17> Works fine "everywhere else" ...
[15:38] <RoyK> as in accessed
[15:39] <robert17> hm. Worth a try ... doing so now.
[15:48] <robert17> RoyK: Completely unneeded on Ubu 16 -- autoassembly on boot works with no access/mount, just mdadm.conf config.  OTOH, on Ubu 14, no autoassembly on boot, UNTIL ... create an /etc/fstab mount for the array.  Works now.
[15:48] <robert17> IIUC, that's going to be a problem if I want to boot from the array ...
[15:49] <robert17> It's got to assemble in order to GET to the /etc/fstab ... unless there's sufficient info in the initramfs
[15:53] <RoyK> robert17: I've used raid1 for boot devices on most versions of debian/ubuntu, and never seen an issue with that
[16:08] <RoyK> robert17: As I said, it usually assembles correctly, but isn't activated unless it's being touched by lvm or a filesystem or similar
[16:10] <robert17> RoyK: Do you by anychance need to use "dolvm" or "lvmwait=" in grub cmd line?
[16:16] <RoyK> robert17: IIRC I've used lvwait once, but I don't remember the occation
[16:35] <robert17> RoyK: fwiw, setting "GRUB_CMDLINE_LINUX="dolvm lvmwait=/dev/mapper/VG_TEST-LV_TEST", where /dev/VG_TEST/LV_TEST is a VG+LV on PV=/dev/md1 seems to do the trick. No /etc/fstab mount or /etc/rc.local needed.  Of course, that still requires that there's a mountable FS somewhere on the array ...
[17:17] <Braven> hello
[17:18] <Braven> does anyone know how to prevent a IF that is configured for DHCP from registering in DNS
[17:19] <Freze> thanks Royk
[17:27] <Braven> ??
[18:15] <Guest8335> What is needed for the broadcom WiFi to work?
[18:24] <madprops> anyone has compiled apache's mod_wsgi for python3 on xenial?
[18:24] <madprops> the version in the repo has a bug
[18:34] <sarnold> madprops: what bug number?
[18:35] <madprops> https://bugzilla.redhat.com/show_bug.cgi?id=1345725
[18:45] <sarnold> madprops: you could download the patch from the github issue; apt-get source the mod_wsgi package, apt-get build-depends for the mod_wsgi package, and try applying the patch
[18:45] <sarnold> looks like our packages are quite a bit older than the patch on github, it might take some massaging, but it's worth a try
[18:46] <sarnold> madprops: and if it works for you, you could post a debdiff to the debian bug and ubuntu bug and see if you can get it fixed for everyone else, too :)
[19:03] <Guest8335> Expansem
[19:04] <sarnold> Guest8335: that's not a very good password.. hope it's not for anything important :)
[19:05] <Guest8335> Sarnold easy to say.
[19:07] <Guest8335> How can I find a kmod name?
[19:08] <Guest8335> Bcmwl-kernel-source looked to have built a module.
[19:08] <Guest8335> It appears it isn't loaded.
[19:09] <sarnold> Guest8335: it looks like that generates a file named wl.ko
[19:10] <sarnold> Guest8335: try running 'find /lib/ -name wl.ko -ls' and see if that finds anything
[19:12] <Guest8335> There was b43 it isn't a 43xx chip but it works on fedora.
[19:20] <Guest8335> sarnold: Whatever it built there's no recent wl .ko showing in /lib/
[19:21] <Guest8335> When the package was installed it looked like it built against the running kernel. Running make in use SRC doesn't work.
[19:25] <sarnold> Guest8335: how about 'dkms status'?
[19:29] <Guest8335> Sarnold status?
[19:29] <Guest8335> It was installed it looks like something to. Do with virtualbox.
[19:58] <inkata> rbasak: hi hi
[20:08] <inkata> .
[20:18] <jamespage> coreycb, hey I sycned nova 14.0.1 to proposed
[20:18] <coreycb> jamespage, thanks
[20:43] <bc2946088> Probably a silly question, but upgrading a 14.04.05 MAAS(1.9) that controls openstack nodes to the latest HWE for continued support shouldn't affect anything with MAAS, right?
[21:00] <roaksoax> bc2946088: what do you mean by to the latest HWE ?
[21:19] <bc2946088> roaksoax: using the 16.04 Xenial HWE Stack on 14.04
[22:06] <jamespage> coreycb, release note looks good
[22:18] <BrianBlaze420> I am trying to set up an openvpn server
[22:18] <BrianBlaze420> oh it came to me
[22:18] <BrianBlaze420> <3
[22:45] <minx> Hi anyone with experience setting up ubuntu servers on lunanode vps?


[01:49] <lduros> how can I remove a remote branch?
[01:50] <lduros> I tried bzr remove-tree ........
[01:50] <lduros> no luck
[01:59] <lduros> rmbranch :-)
[02:09] <lduros> now i'd like to do the same for an entire repo
[14:37] <BasicOSX> haven't seen a update to bzr.dev in awhile did the repo move? I've been at revision 6523 for several weeks.
[14:58] <jelmer> BasicOSX: no, the repository is still in the same location
[15:19] <BasicOSX> Did poolie take a job at Google?
[20:08] <lifeless> wgz: around ?
[20:08] <lifeless> wgz: I has a python3 IO question:
[20:08] <lifeless> :!python3 -c "import sys; print(type(sys.stdin.read()))"
[20:08] <lifeless> <class 'str'>
[20:09] <lifeless> in subunit, we have a _make_stream_binary that you wrote
[20:09] <lifeless> on Python3 its a no-op, but ... we still need it
[20:09] <lifeless> to do something here
[20:11] <lifeless> is pulling out .buffer 'ok' ?


[01:10] <mclemenceau>  /buffer 7
[11:09] <Fantu> hi, freeipmi migration from proposed is blocked by an autotestpkg regression (https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#freeipmi) anyway on debian I not saw it and from a fast look to log I don't understand if there is a real issue, I tried to retry the failed test but tell I don't have the permission, can someone try to retry it please?
[11:44] <ginggs> Fantu: .
[11:48] <Fantu> ginggs thanks for reply, "." stay for what?
[12:11] <ginggs> Fantu: i retried the autopkgtest
[12:14] <ginggs> but it seems to have failed again
[16:53] <ahasenack> waveform: hi, do I need both the new debhelper and debconf?
[17:10] <waveform> ahasenack, no just the debconf -- the debhelper fix is just for the building of the package itself (i.e. it affects the prerm, preinst, etc. of the slapd package but doesn't interact with the running of those scripts)
[17:11] <ahasenack> I saw a note in the bug that debhelper was just uploaded, but I thought it had landed already
[17:11] <ahasenack> I guess it migrated, ah
[17:11] <waveform> yup -- debconf contains dpkg-reconfigure and defines which of the maintainer scripts are run during dpkg-reconfigure (this previously excluded preinst, which is now important after the fix in debhelper)
[17:11] <ahasenack> waveform: so I don't need to rebuild it with the new debconf, just try it again with the new debconf?
[17:13] <waveform> nope, no need to rebuild -- just install debconf (to get the updated dpkg-reconfigure) and see if that does the trick
[17:16] <ahasenack> waveform: it worked, I updaded the bug
[17:16] <waveform> ahasenack, lovely! I'll stick a debdiff on there and see if I can find a sponsor
[17:16] <ahasenack> thx
[18:05] <ginggs> waveform: i'll take a look
[18:06] <waveform> ginggs, thanks!
[18:26] <ginggs> waveform: sponsored
[21:08] <roehling> is there a way to retrigger a failed build (in particular, the s390x build of filament) if the failure was fixed by an updated dependency (in this case, glslang)?
[21:45] <Eickmeyer> roehling: You should be able to trigger a rebuild on Launchpad if the dependency is updated.
[21:46] <Eickmeyer> roehling: I just retriggered the build for you. :)
[21:46] <roehling> Thank you :)


[06:47] <mlankhorst> morning
[09:38] <xnox> when i swtich to tty1: vga monitor brightness drops, dvi monitor stays the same. annoying =) how can I fix this?
[14:20]  * apw is dropping to low-resolution mode, though i can see nothing in the Xorg.0.log to say why, it just quits
[14:32] <mlankhorst> xnox: probably a monitor thing then :P
[14:32] <mlankhorst> apw: hm attach debugger? if it's a case of a missing symbol xorg will just quit without reporting anything..
[14:33] <xnox> mlankhorst: but, but my dual screen monitors are the same: vga->vga goes dim, hdmi->dvi does not. And it's clearly os driven, e.g. focusing on a Virtual Machine KVM on the vga screen makes it go dim, moving that window to dvi screen makes the vga go bright again.
[14:33] <xnox> it's a desktop, yet the vga screen behaves as if it's a laptop one with battery saving stuff.
[14:34] <apw> mlankhorst, hmmm i see that is unhelpful
[14:35] <xnox> mlankhorst: thanks for the hint. the settings on the screens are different depending on the input. Disable "active auto contrast" on the vga and all seems to be fine =)
[14:37] <apw> mlankhorst, it also seems to be a race of some kind, as it does not occur on all boots
[14:38] <mlankhorst> apw: fun :/
[14:38] <apw> mlankhorst, i know
[14:39] <mlankhorst> I tried putting the nv50 in my computer and was hitting a similar issue, now I'm putting the nv50 in an old computer with a serial port, so that I can bisect it, so that I can use it in my own computer, so that I can bisect the nvc0 locking up on mmiotrace :/
[14:43] <apw> mlankhorst, nice
[14:43] <apw> (in a tecno-purv kind of way)


[12:23] <mrevell> #t/nick mrevell-lunch
[12:23] <mrevell> damn
[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <sinzui> me
[16:00] <bigjools> me
[16:00] <matsubara> so, who is here today?
[16:00] <matsubara> me
[16:01] <matsubara> flacoste, Ursinha, Rinchen,?
[16:01] <flacoste> me
[16:01] <Ursinha> me
[16:01] <bigjools> cprov, ping
[16:01] <cprov> pong
[16:01] <cprov> me
[16:01] <herb> me
[16:02] <matsubara> we're missing, code
[16:02] <matsubara> rockstar: ?
[16:02] <intellectronica> me
[16:02] <flacoste> i'm Foundations and the DBA report
[16:02] <matsubara> Ursinha: who's the qa contact for rosetta?
[16:02] <matsubara> danilo[home]__: ?
[16:02] <danilos> me
[16:02] <Ursinha> danilo[home]__, ping
[16:03] <matsubara> ah hinding there
[16:03] <matsubara> ok, so we're missing code
[16:03] <danilos> matsubara: right, danilo[home]__ is, surprisingly, my account at home :)
[16:03] <matsubara> let's move on then
[16:04] <matsubara> [TOPIC] Agenda
[16:04] <MootBot> New Topic:  Agenda
[16:04] <matsubara>  * Next meeting
[16:04] <matsubara>  * Actions from last meeting
[16:04] <matsubara>  * Oops report & Critical Bugs
[16:04] <matsubara>  * Operations report (mthaddon/herb/spm)
[16:04] <matsubara>  * DBA report (DBA contact)
[16:04] <matsubara>  * Sysadmin requests (Rinchen)
[16:04] <matsubara> [TOPIC] Next meeting
[16:04] <MootBot> New Topic:  Next meeting
[16:04] <matsubara> next meeting, same time next week? ok for everyone?
[16:04] <flacoste> yep
[16:04] <Rinchen> me
[16:04] <rockstar> matsubara, here
[16:04] <flacoste> stub should be able to make it
[16:05] <matsubara> Rinchen, rockstar: noted. thanks
[16:05] <flacoste> he had a previous engagement tonight
[16:05] <Ursinha> flacoste, nice
[16:05] <matsubara> great
[16:05] <rockstar> I forgot that we were doing this earlier.
[16:05] <matsubara> [TOPIC] * Actions from last meeting
[16:05] <MootBot> New Topic:  * Actions from last meeting
[16:05] <matsubara>  * stub to patch our fti regexp to avoid OOPSes (bug 174368) and discuss a proper fix with jtv
[16:05] <flacoste> matsubara: that's not started
[16:06] <Ursinha> a hanging bug
[16:06] <flacoste> honestly, i don't think it's high priority
[16:06] <matsubara> hmm shall I keep bring it up during this meeting?
[16:06] <flacoste> it mostly trigger an OOPS when somebody tries spamming one of our search fields
[16:06] <flacoste> well deserved imho
[16:06] <flacoste> :-)
[16:07] <Ursinha> :)
[16:07] <matsubara> :-)
[16:07] <Ursinha> matsubara, well, guess not so
[16:07] <flacoste> but you might have a different opinion?
[16:07] <matsubara> I'd like to have it targeted to a milestone at least
[16:07] <Ursinha> yes
[16:07] <matsubara> so we won't keep pushing oops bugs to the end of the queue
[16:07] <bigjools> well, if it's acceptable it should not be an OOPS should it?
[16:08] <flacoste> it's not acceptable
[16:08] <flacoste> there is some rare, but legitimate query that are affected by it
[16:08] <matsubara> it's important that we fix OOPS bugs, even if they affect a few users
[16:08] <danilos> targeting to a milestone doesn't mean much if there is no real dedication to finish it in a certain timeframe
[16:08] <flacoste> yepp
[16:09] <Ursinha> danilos, indeed
[16:09] <danilos> some bugs, like this one, for example, are not known how much they might take to solve (it might be a bunch of different things)
[16:09] <flacoste> i'll keep it on our radar
[16:09] <matsubara> well, that's the point of targeting it, isn't it? you set a deadline to fix it
[16:09] <flacoste> and try to get to it at the end of the cycle
[16:09] <matsubara> all right. thanks flacoste
[16:09] <matsubara> I take it off from the actions from last meeting
[16:10] <matsubara> [TOPIC] * Oops report & Critical Bugs
[16:10] <MootBot> New Topic:  * Oops report & Critical Bugs
[16:10] <matsubara> Today's oops report is about bugs 271561, 273363
[16:10] <matsubara> rockstar, any news about #271561?
[16:11] <matsubara> that's been happening at least once a day and I didn't see any progress in the bug report.
[16:11] <rockstar> matsubara, it's being worked on, that's all I know about it.
[16:11] <matsubara> flacoste, do you think bug 273363 might be related to bug 271902?
[16:11] <sinzui> Edwin has a fix for tno attribute 'read_only' I beleive
[16:11] <flacoste> matsubara: stub has a fix in review for that one
[16:11] <flacoste> and yes, they are dupped
[16:12] <flacoste> sinzui?
[16:12] <matsubara> sinzui and flacoste you might want to coordinate who will fix it then? :-)
[16:12] <flacoste> well, it's assigned to stuart
[16:12] <matsubara> but I guess that's on stub's turf
[16:12] <sinzui> 273363 is assigned to Edwin
[16:12] <flacoste> lol
[16:12] <matsubara> the other one is assigned to EdwinGrubb
[16:12] <flacoste> well, stuart has a branch fixing both in review
[16:13] <sinzui> It is caused by Edwins fix to the cookie issue with feeds
[16:13] <flacoste> i'll speak to Edwin
[16:13] <matsubara> rockstar: can you assign it to the devel fixing that issue and change the status to in progress?
[16:13] <flacoste> matsubara: can you dup it?
[16:13] <matsubara> flacoste: sure
[16:13] <rockstar> matsubara, sure.
[16:13] <matsubara> thanks guys.
[16:13] <Ursinha> ok
[16:13] <matsubara> Ursinha: stage is yours
[16:14] <Ursinha> one critical, bug 273489
[16:14] <Ursinha> danilos, i've sent one email to jtv yesterday, to get more details on the problem
[16:14] <danilos> right
[16:14] <Ursinha> can you help me with that after the meeting?
[16:14] <danilos> this has basically been 'fix committed'
[16:14] <danilos> we are now importing all the Intrepid templates
[16:14] <Ursinha> right
[16:14] <matsubara> I also have one soyuz critical as well. yesterday cprov identified an oops that affected 60 or so PPA's
[16:15] <danilos> and this morning there were around 13K files left to import (yesterday afternoon around 18K)
[16:15] <danilos> so, this should be completely fixed in two days at most
[16:15] <matsubara> cprov: can we expect a IR for that one? I presume no bug was filed and things were already fixed, right?
[16:15] <Ursinha> danilos, great, thanks
[16:15] <cprov> matsubara: and it was fixed at that time as well, which a production update query.
[16:15] <bigjools> it was only edge that was affected
[16:15] <Ursinha> matsubara, which bug is it?
[16:15] <matsubara> so, no code change needed?
[16:15] <cprov> matsubara: no
[16:16] <danilos> Ursinha: ping me after the meeting for more details, but right after the meeting, I am likely to get out soon
[16:16] <matsubara> Ursinha: no bug reported for that one
[16:16] <cprov> matsubara: as I said, we just had to rush the data migration.
[16:16] <matsubara> cprov: ok. so, Rinchen asked about doing an IR for that.
[16:17] <Ursinha> matsubara, are we going to file one?
[16:17] <Rinchen> Did file a critical bug for that?
[16:17] <Rinchen> so, if we didn't file a bug....
[16:17] <matsubara> no, I can file one, but it's kinda pointless, isn't it? since the problem is fixed
[16:18] <Rinchen> matsubara, only in the sense that we fixed the problem, not what caused the problem
[16:18] <matsubara> and there'll be no code change
[16:18] <bigjools> it's not a bug, it's an edge rollout issue
[16:18] <Rinchen> so let's start please with a write up to the dev list about what happened and how to prevent it and not do an IR
[16:18] <Rinchen> is that acceptable to everyone?
[16:18] <bigjools> cprov already did that last night
[16:18] <cprov> Rinchen: yes, email already sent.
[16:19] <Rinchen> ok, thanks. I've been searching for it but haven't found it yet. :-(
[16:19] <Rinchen> I'll keep looking.
[16:19] <Rinchen> Thanks!
[16:19] <Rinchen> and thanks for resolving it quickly last nigiht
[16:19] <matsubara> thanks
[16:20] <matsubara> moving on
[16:20] <Ursinha> thanks guys
[16:20] <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
[16:20] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[16:20] <herb> * 2008-09-19 - Updated 2.1.9 to r7035. This update included planned downtime. The service was down for approximately an hour.
[16:20] <herb> * During the week we've had a few app servers die and leave core files. flacoste was investigating.
[16:20] <herb> * 2008-09-23 - Cherry pick r7058 and r7064 to the scripts server and bzrsyncd server respectively.
[16:20] <herb> * 2008-09-24 - Cherry pick r7066 and r7072 to lpnet*, update edge* to r7072.
[16:20] <flacoste> herb: so i investigated the core files
[16:20] <herb> flacoste: any update on the dying app servers?
[16:20] <herb> cool
[16:21] <flacoste> unfortunately, the stack track is pretty useless
[16:21] <herb> boo
[16:21] <flacoste> seems like accessing corrupted memory
[16:21] <flacoste> things that barry and mwhudson said we could consider is running the appserver
[16:21] <flacoste> using python2.4-dbg
[16:21] <flacoste> which has some more debugging stuff in it
[16:21] <flacoste> but it requires that the packages we are using also have a -dbg build
[16:21] <flacoste> and are three times slower
[16:22] <flacoste> another interesting thing
[16:22] <herb> that's less than ideal.
[16:22] <flacoste> is the stack trace posted by jtv
[16:22] <flacoste> that occured in one of his script
[16:22] <flacoste> it seems to point to a zope or storm problem
[16:22] <flacoste> in the case of zope, the landscape team has a fixed for it
[16:23] <flacoste> we should get it by moving to zope 3.4 (which we are going to attempt next week)
[16:23] <flacoste> but it wasn't clear from the discussion i overheard if it looked like the same problem they experienced
[16:23] <flacoste> another hypothesis made it a problem with the Storm C extensions
[16:24] <flacoste> it's a good working hypothesis that the script and app server death is related to the same symptom
[16:24] <flacoste> the fact that we have a better stack trace in the script case is probably due to the fact that it runs single-threaded
[16:24] <herb> ok. short of running with all -dbg packages is there anything we can do to help isolate the problem?
[16:24] <flacoste> so next step is to follow-up on the jtv, gustavo, barry discussion
[16:25] <flacoste> and see what conclusions was there
[16:25] <flacoste> and if there is something we can try from there
[16:25] <flacoste> one last thing
[16:25] <flacoste> i gave mthaddon the command to extract a stacktrace from a core file
[16:25] <flacoste> that's really the only thing we need or can do with it
[16:25] <mthaddon> https://launchpad.canonical.com/OSA/HowTo/BacktraceFromCoredump
[16:25] <flacoste> so you could just save the stack trace instead of the whole core files
[16:25] <flacoste> mthaddon: awesome!
[16:25] <herb> flacoste: ok. cool.
[16:26] <flacoste> EOT unless there are questions
[16:26] <herb> that's it from the LOSAs unless there are any questions.
[16:26] <matsubara> cool. thanks for the thoroughly explanation flacoste
[16:26] <matsubara> and thanks herb, mthaddon!
[16:26] <matsubara> [TOPIC] * DBA report (DBA contact)
[16:26] <MootBot> New Topic:  * DBA report (DBA contact)
[16:26] <flacoste> Nothing unusual I'm aware of on the production database.
[16:26] <flacoste> Replication testing scheduled this week using demo.launchpad.net as
[16:26] <flacoste> per discussions with Francis. Turn off the monitoring if it was
[16:27] <flacoste> switched back on.
[16:27] <flacoste> Assuming demo.launchpad.net testing doesn't push us back to the
[16:27] <flacoste> drawing board, I want to have a replication version of the  staging
[16:27] <flacoste> rollout scripts ready for next cycle, which should involve some
[16:27] <flacoste> testing this cycle to ensure they actually work.
[16:27] <flacoste> Schedule is to have the production Launchpad database replicated as
[16:27] <flacoste> part of the 2.1.11 release. Staging running replicated for the whole
[16:27] <flacoste> cycle should give enough experience for signoff from everyone. There
[16:27] <flacoste> should be no unusual downtime requirements. I will need to be around
[16:27] <flacoste> for the rollout though.
[16:27] <flacoste> The new DB baseline doesn't affect production or staging.
[16:27] <flacoste> mthaddon, herb: stuart will send you over notes on the changes this means for the staging roll-out process
[16:27] <flacoste> i can take questions
[16:28] <mthaddon> flacoste, great, thx
[16:28] <herb> flacoste: ok.  when should we expect notes?
[16:28] <flacoste> not before the end of next week i think
[16:28] <flacoste> it will take that much time to test on demo
[16:28] <flacoste> and then upgrade the staging scripts
[16:28] <herb> flacoste: at a high level how will this change the rollout process?
[16:28] <matsubara> flacoste: can Ursinha and I help with the testing somehow?
[16:28] <flacoste> the restoration of the DB
[16:29] <flacoste> matsubara: i don't think so, at this stage it's not about QA, but more about seeing performance
[16:29] <flacoste> matsubara: i'll talk to stuart and forward the offer, i might be mistaken
[16:29] <flacoste> herb: we'll be having a replicated DB (so a master and a slave) on staging
[16:30] <matsubara> flacoste: okie. tell him to ping/email us if he needs something
[16:30] <flacoste> so this affects how the DB sync is done
[16:30] <herb> flacoste: ok
[16:30] <herb> flacoste: thanks
[16:30] <matsubara> all right. thanks flacoste
[16:30] <matsubara> [TOPIC] * Sysadmin requests (Rinchen)
[16:30] <MootBot> New Topic:  * Sysadmin requests (Rinchen)
[16:30] <Rinchen> Hi!
[16:30] <Rinchen> Is anyone blocked on an RT or have any that are becoming urgent?
[16:31] <matsubara> I have one RT #31795, which I'd like to suggest priority ~80
[16:31] <Rinchen> Does anyone thing this section of the meeting is worthwhile?
[16:31] <rockstar> Rinchen, it might if me had an RT...
[16:31] <Rinchen> :-)
[16:31] <rockstar> s/me/we
[16:31] <danilos> Rinchen: I guess we are free to raise RT tickets with you and our team leads whenever we want anyway
[16:31] <bigjools> I think it is worthwhile
[16:31] <Rinchen> matsubara, ok, will look into that
[16:31] <intellectronica> maybe we should make it possible for people to add an RT-related item to the agenda before the meeting, if they have one, but skip the section if no-one does
[16:31] <matsubara> Rinchen: thank you
[16:31] <bigjools> if nothing else it acts as a memory jog
[16:32] <danilos> intellectronica: +1
[16:32] <matsubara> intellectronica: I like that
[16:32] <Rinchen> danilos, intellectronica - yeah, that was my point.  If others find it helpful though, I'm happy to continue it.
[16:32] <Rinchen> it's not like it's a hard thing to do :-D
[16:32] <Rinchen> I'll let matsubara and Ursinha make the call.
[16:32] <Rinchen> Any other tickets?
[16:33] <Rinchen> ok, if you do have something, please ping me!
[16:33] <Rinchen> thanks matsubara
[16:33] <matsubara> ok, let's experiment with intellectronica suggestion for awhile then. I'll update MeetingAgenda page to reflect that
[16:33] <matsubara> thank you Rinchen
[16:34] <matsubara> anything else before I close?
[16:34] <matsubara> all right
[16:34] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[16:35] <matsubara> actually the log part is a lie
[16:35] <flacoste> thzx!
[16:35] <matsubara> but thanks!
[16:35] <matsubara> #endmeeting
[16:35] <MootBot> Meeting finished at 10:35.
[16:35] <danilos> thanks all, matsubara especially for running the meeting :)
[16:36] <matsubara> np
[16:36] <intellectronica> thanks, matsubara
[16:36] <Ursinha> thanks matsubara