[01:46] <davecheney> thumper: more debug is neede here
[01:46] <davecheney> 2013-08-29 01:42:25 DEBUG juju.worker.uniter.filter filter.go:289 got unit change
[01:46] <davecheney> 2013-08-29 01:42:26 INFO juju.worker.upgrader upgrader.go:138 required tools: 1.13.2-precise-amd64
[01:46] <davecheney> all the units are blocked on this line
[01:46] <davecheney> no further output
[01:46] <thumper> hmm...
[01:47] <thumper> doesn't seem too helpful
[01:47] <davecheney> this is in ap-southeast-2 as well
[01:47] <bigjools> wallyworld__: how's the coffee machine?
[01:47] <davecheney> so it shouldn't take more than a few seconds to get the tools
[01:49] <thumper> I wonder where it is blocked
[01:50] <davecheney> i'll hit it with SIGQUIT and hope stderr goes somewhere
[01:51] <axw> wallyworld__: I've just pushed some changes to my image-metadata branch that fixes the marshalling so numbers aren't floats
[01:51] <davecheney> thumper: http://paste.ubuntu.com/6038698/
[01:52] <axw> wallyworld__: it's a bit gnarly though, you might want to review my changes to environs/simplestreams
[01:52] <davecheney> thumper: right, it did the upgrade, but it didn't restart
[01:53] <thumper> hmm...
[01:54] <davecheney> hmm, maybe it worked
[01:55] <davecheney> hard to tell from the output
[01:59] <davecheney> thumper: ok, here is the issue
[01:59] <davecheney> 2013-08-29 01:59:40 INFO juju runner.go:253 worker: start "upgrader"
[01:59] <davecheney> 2013-08-29 01:59:43 DEBUG juju.worker.uniter.filter filter.go:289 got unit change
[02:00] <davecheney> 2013-08-29 01:59:43 INFO juju.worker.upgrader upgrader.go:138 required tools: <IP_ADDRESS>-precise-amd64
[02:00] <davecheney> ^ this is the message from the upgrade _after_ it has restated
[02:04] <davecheney> oops 2013-08-29 02:03:48 INFO juju.worker.uniter context.go:234 HOOK Shutting down without a db
[02:04] <davecheney> 2013-08-29 02:03:48 INFO juju.worker.uniter context.go:234 HOOK /var/lib/juju/agents/unit-mediawiki-0/charm/hooks/db-relation-departed: line 4: /var/lib/juju/agents/unit-mediawiki-0/charm/hooks/stop: No such file or directory
[02:04] <davecheney> nope, charm bug
[02:05] <wallyworld__> bigjools: awesome. i've just made a second one for today
[02:05] <bigjools> wallyworld__: \o/
[02:05] <wallyworld__> axw: thanks, i'll take a look
[02:06] <axw> wallyworld__: thanks. also, I started using tools.Fetch to test... but it looks like that only returns a single version? is that right?
[02:07] <axw> so I guess I'll just grab the files from storage and check their contents directly
[02:08] <wallyworld__> axw: the Fetch method can return multiple but the constraint limits it. i've pushed a branch for review which allows loser matching and hence returns multiple
[02:08] <axw> wallyworld__: ok, cool
[02:08] <wallyworld__> to generate the metadata, i think the current best option is to grab from storage
[02:08] <wallyworld__> we will/should revist that i think
[02:24] <wallyworld__> axw: it is hacky isn't it. perhaps we can make it better by working the construct logic into the unmarshall method
[02:24] <axw> wallyworld__: moving it into which unmarshal method?
[02:25] <wallyworld__> the json unmarshaller for the collection
[02:26] <wallyworld__> so instead of shoving stuff into a "" key, we can call construct and fill out the collection directly
[02:26] <axw> wallyworld__: the problem is that the item type isn't known there
[02:27] <axw> wallyworld__: there's no way to convey context to the unmarshaller, except through a global :(
[02:27] <wallyworld__> is it sort of since we figure out the call point
[02:27] <axw> hm
[02:27] <wallyworld__> so we can map that to type
[02:27] <axw> I don't like any of the solutions :)
[02:27] <wallyworld__> i'm being a bt hand wavey, but i think it is possible
[02:28] <wallyworld__> i agree, they all suck
[02:28] <axw> yeah I get you
[02:28] <wallyworld__> i think Go's json unmarshalling sucks
[02:28] <wallyworld__> it can't be extended quite right
[02:29] <wallyworld__> at least we would be confining the hackery to a single unmarshall method
[02:29] <wallyworld__> so the method is a black box that "just works" but is hacky inside
[02:29] <wallyworld__> and i'd put it separately in a json.go file in the same package
[02:31] <axw> hmmmm
[02:31] <axw> I'll have a look later
[02:31] <wallyworld__> ok
[02:31] <axw> need to respond to some review comments
[02:31] <wallyworld__> np
[02:34] <axw> wallyworld__: so one way you could do it is this: have ParseCloudMetadata store itemType in a global, protected with a mutex (only one unmarshal at a time); have ItemsMap.UnmarshalJSON do the Callers check, and then use the global
[02:34] <axw> is that what you were thinking?
[02:35] <wallyworld__> axw: sort of. i was thinking there'd be a map of method name (as determined in the current code) -> type
[02:35] <wallyworld__> so no mutex needed
[02:35] <wallyworld__> just look u the map after calling method is determined
[02:36] <axw> ah yeah ok
[02:36] <axw> so they just register up front the function name -> reflect.Type
[02:37] <wallyworld__> yep
[02:37] <wallyworld__> still hacky, but it allows it to be kept under the rug
[02:38] <wallyworld__> and isolated from the core business logic
[03:23] <axw> thumper: can you please explain this to me (comment by William)? "Never use conn.Environ if you can possibly help it. It's basically never up to date."
[03:24] <thumper> axw: sure
[03:24] <thumper> the conn.Environ is the environment based purely on the parsing of the local config
[03:24] <thumper> not the value in the bootstrap node
[03:24] <thumper> when a machine is bootstrapped
[03:24] <thumper> is initializes passwords etc
[03:24] <thumper> so they are no longer the same
[03:24] <axw> ah right
[03:24] <thumper> also, if someone calls juju set-environment
[03:24] <thumper> it doesn't modify the local
[03:25] <thumper> only the bootstrap copy
[03:25] <thumper> which then notifies all the workers
[03:25] <thumper> does that make sense?
[03:25] <axw> thumper: yes
[03:25] <thumper> cool
[03:26] <axw> I've seen the code that updates config from state
[03:26] <axw> not sure about an Environ tho
[03:27] <davecheney> trollololo, more bugs
[03:33] <davecheney> https://bugs.launchpad.net/juju-core/+bug/1218168
[03:33] <_mup_> Bug #1218168: cmd/juju: upgrade-charm does not expand tilde in filepaths <papercut> <juju-core:Triaged> <https://launchpad.net/bugs/1218168>
[04:08]  * thumper runs the tests knowing that they'll fail
[04:09]  * thumper was very surprised to see them all pass
[04:09] <thumper> was in the wrong pipe
[04:12]  * thumper smiles at the failures
[04:12] <thumper> seing base64 encoded certs rather than yaml serialized []byte
[04:34] <davecheney> thumper: so you fixed that huge turd of output in the cloud-init-output ?
[04:42] <davecheney> bigjools: https://bugs.launchpad.net/maas/+bug/1218182
[04:42] <_mup_> Bug #1218182: No way to put a node into "maintenance mode" <MAAS:New> <https://launchpad.net/bugs/1218182>
[04:42] <davecheney> how can this be a thing
[04:43] <davecheney> surely someone has asked for this already
[04:43] <bigjools> just re-commission it
[04:43] <bigjools> but yes generally that would be useful
[04:43] <bigjools> we even have a node state for it that's not used yet
[04:52] <davecheney> bigjools: can you recomission remotely ?
[04:52] <davecheney> or does one need to get off ones ass ?
[04:53] <bigjools> davecheney: just click on the ui button to do it
[04:53] <bigjools> however
[04:53] <bigjools> it'll do the usual cycle and may not fail if nothing is getting installed
[04:53] <bigjools> we need to add commissioning tests
[04:54] <bigjools> deletion is the best way of taking it out for now
[04:58] <jam1> davecheney: ~ is expanded by your shell, and has different behaviors if you attach an argument. Try: "echo -f=~/" vs "echo -f ~/" For me the former doesn't expand, the latter does expand.
[05:00] <davecheney> jam: we hit this in japan
[05:00] <davecheney> it would be nice if there was a solution
[05:01] <jam> davecheney: use --repository ~/foo/bar
[05:01] <jam> it works
[05:01] <jam> don't use --repository=~/foo/bar it doesn't
[05:01] <davecheney> lucky(~/src/launchpad.net/juju-core) % juju upgrade-charm --repository=~/charms --switch local:mediawiki mediawiki
[05:01] <jam> bash-ism
[05:01] <davecheney> error: no repository found at "/home/dfc/src/launchpad.net/juju-core/~/charms"
[05:01] <davecheney> sure
[05:01] <davecheney> but can we expand it in the command
[05:30] <axw> thumper: did you want to do another review of my manual provisioning changes, or are you happy with it? fwereade_ has LGTM'd
[05:41] <thumper> axw: if you don't mind, I'll take a quick look
[05:41] <thumper> but probably just before the meeting
[05:42]  * thumper breaks for dinner, back later
[05:45] <axw> thumper-dinner: not at all, I will go back to simplestreams stuff for now
[05:46] <axw> jam: what is that gwacl/failing-test thing about?
[05:47] <jam> axw: sorry for the noise. bigjools has asked that I set up gwacl under our tarmac bot, and I want to test that it both "successfully lands good patches" and "successfully fails bad patches"
[05:49] <axw> jam: I was just curious, not bothered :)
[05:49] <axw> thanks
[06:23] <rogpeppe1> mornin' all
[06:28] <jam> morning rogpeppe1
[06:28] <rogpeppe> jam: hiya
[06:33] <rogpeppe> jam: any idea what might be going on here? It stopped my merge last night and i can't reproduce it locally. https://code.launchpad.net/~rogpeppe/juju-core/376-factor-out-provider-utils/+merge/182465/comments/414178
[06:33] <rogpeppe> fwereade_: ^
[06:35] <jam> rogpeppe: the only thing that comes to mind is that the test suite is "leaking" information an the test case is trying to contact a mongodb running elsewhere.
[06:35] <jam> (either a 'zombie' one that didn't get torn down properly or something else)
[06:36] <rogpeppe> jam: that is a possibility i suppose, if the port selection logic isn't working
[06:36] <rogpeppe> jam: i'm considering changing it actually
[06:37] <rogpeppe> jam: currently it makes a socket, letting the system choose the port, then closes the socket, and trusts that it will be still ok to use in a short while
[06:37] <rogpeppe> jam: we can't avoid some window, but i'm wondering if it might be better to pick a port at random, check that we can't dial it, then use that
[06:38]  * rogpeppe wishes the port name space was considerably larger
[06:38] <jam> rogpeppe: could we pick a port but open it as reopenable and only close it after something else has grabbed it?
[06:39] <jam> SOCK_REUSEADDR or whatever that param is
[06:39] <rogpeppe> jam: i don't think SOCK_REUSEADDR works like that
[06:39] <rogpeppe> jam: it only allows address reuse of unique local/remote pairs, AFAIR
[06:39] <rogpeppe> jam: i don't think it allows you to bind two listeners to the same port
[06:39] <jam> http://stackoverflow.com/questions/775638/using-so-reuseaddr-what-happens-to-previously-open-socket
[06:40] <jam> rogpeppe: probably
[06:40] <jam> I know you can bind them to the same port by doing fork magic
[06:40] <rogpeppe> jam: you mean by inheriting the fd?
[06:40] <jam> rogpeppe: right. it is how apache used to do it with their forking daemons.
[06:41] <jam> one of the subprocesses "wins" the Accept request
[06:41] <rogpeppe> jam: that's kind of a different thing - it's just sharing the already bound socket
[06:41] <jam> rogpeppe: though to be fair, I don't have high expectations that it is actually the bug you are seeing.
[06:41] <jam> rogpeppe: but I *have* been seeing a lot of zombie mongodb's this week.
[06:41] <rogpeppe> jam: i went through a rash of "cannot bind to port" problems last week
[06:43] <jam> rogpeppe: "SO_REUSEPORT... allows you to bind an arbitrary number of sockets to exactly the same source address and port as long as all prior bound sockets also had SO_REUSEPORT"
[06:43]  * rogpeppe didn't know about REUSEPORT
[06:43] <jam> but it looks like it may be a BSD flag
[06:43] <rogpeppe> jam: but i don't think that helps us
[06:44] <rogpeppe> jam: even if it was available
[06:44] <rogpeppe> jam: because mongod won't be using that flag
[06:44] <jam> rogpeppe: "Linux 3.9 added the option SO_REUSEPORT to Linux as well"
[06:44] <jam> rogpeppe: it does
[06:44] <jam> because *we* use that flag
[06:44] <jam> and then the next person can bind without the flag
[06:44] <jam> at least from what I read
[06:44] <jam> I could be wrong
[06:45] <rogpeppe> jam: i slightly doubt it. let's check, one mo
[06:45] <jam> rogpeppe: the wording is a bit funny, so no guarantees
[06:45] <jam> also, I think precise is older than kernel 3.9
[06:45] <rogpeppe> jam: the wording sounds to me as if all binders to the port must use that option
[06:45] <rogpeppe> jam: otherwise it's really quite dangerous
[06:46] <rogpeppe> jam: because i might wish to bind a server to a port, but because the previous server on that port has used REUSEPORT, it allows us anyway, and then we get two different servers randomly sharing the same network address
[06:47] <rogpeppe> jam: there's another possibility which won't rule out the failure but will make it more obvious what's happening
[06:48] <rogpeppe> jam: which is to get the API server to send back the env UUID and have the client check that
[06:48] <rogpeppe> jam: then at least we will definitively know when we're talking to an unexpected server
[06:50] <jam> rogpeppe: except we've already gotten rejected by that point haven't we? Or are you saying we send it before login?
[06:51] <rogpeppe> jam: we'd need to make it available even to non-logged-in clients though, and i'm not sure if that would be judged to be an unwanted information leak
[06:51] <jam> haven't we validated the cert by this point?
[06:51] <rogpeppe> jam: yeah, we have validated the cert, yes
[06:51] <rogpeppe> jam: and the cert is randomly generated, so actually that's a good point
[06:53] <rogpeppe> jam: and i *think* we use a secure mongo connection even in tests
[06:55] <rogpeppe> jam: no, i don't think it can be a duplicate mongo problem
[06:55] <jam> rogpeppe: I know that if you have a mongo that doesn't support TLS the test suite dies in horrible ways
[06:58] <rogpeppe> hrmph, well it's managed to merge noe
[06:58] <rogpeppe> now
[07:00] <rogpeppe> that failure does concern me though. wtf can be going on, when we've got two concurrent sessions connected to the same API port, the second of which fails logging in with exactly the same creds as the first succeeded with?
[07:05] <rogpeppe> fwereade_: i just noticed this comment of yours: "
[07:05] <rogpeppe> Never use conn.Environ if you can possibly help it. It's basically never up to
[07:05] <rogpeppe> date.
[07:05] <rogpeppe> "
[07:05] <rogpeppe> fwereade_: what does it mean for an Environ to be "up to date"?
[07:06] <rogpeppe> fwereade_: ah, you mean that it might have out of date config attrs?
[07:09] <jam> rogpeppe: he means that conn.Environ is read from local disk, but the source of truth is actually conn.State.Environ
[07:09] <jam> or whatever the actual request is
[07:10] <jam> rogpeppe: which is why for the CLI API stuff we went away from using APIConn and are trying to get away with just api.Client
[07:10] <rogpeppe> jam: yeah
[07:10] <rogpeppe> jam: (well, that's not the only reason for avoiding client use of Environ)
[07:11] <jam> rogpeppe:  this one also failed earlier today, and left a mongodb running: https://code.launchpad.net/~allenap/juju-core/makefile-stuff/+merge/181113
[07:11] <jam> a Watcher didn't err when it exited
[07:11] <jam> I don't know why
[07:11] <jam> I'm concerned we introduced some race conditions recently without realizing it.
[07:11] <jam> Or the bot is just on a VM that is having neighbor issues, which triggers these less frequent problems.
[07:14] <rogpeppe> jam: interesting, that also died in TestManageStateServesAPI
[07:14] <rogpeppe> jam: (the same place i saw the problem)
[07:15] <jam> rogpeppe: I don't quite see that in the 500 line panic, but I trust you did
[07:15] <jam> and yes, we might just have a flakey test, that when it fails might leave a mongodb running.
[07:16] <rogpeppe> jam: (to find which test is running, search for .Test in the stack trace
[07:16] <rogpeppe> )
[07:16] <rogpeppe> jam: even if we have moribund mongodb's, that shouldn't be a problem AFAICS
[07:17] <jam> rogpeppe: I'm saying the test is *causing* moribound mongodb's not that it is affected by them.
[07:17] <rogpeppe> jam: ah, right, yeah
[07:17] <jam> (test suite teardown is known to fail to tear down the mongodb it started)
[07:17] <rogpeppe> jam: it probably happens when a goroutine panics without a recover
[07:18] <rogpeppe> jam: (that happened in this case)
[07:21] <jam> rogpeppe: I thought gocheck recovered from all panics in order to catch them cleanly and report errors from test cases ?
[07:21] <jam> is this happening in a TearDownSuite or something where it isn't being caught?
[07:21] <rogpeppe> jam: it can't recover from panics in goroutines that it didn't start
[07:22] <jam> rogpeppe: ah, even if it starts the one that started it
[07:22] <rogpeppe> jam: which is true in this case - the panic is in a watcher goroutine
[07:22] <rogpeppe> jam: yes - you can't catch panics from many goroutines at once - that would be nastily asynchronous
[07:23] <rogpeppe> jam: the usual solution is to put some cleanup code outside the main executable
[07:27] <axw> wallyworld_: I came up with a better approach to the JSON problem. It's now very similar to how it was originally, but without the float problem
[07:28] <wallyworld_> great :-)
[07:28] <axw> and with no runtime callstack crap
[07:28] <wallyworld_> hooray, i'll take a look in a bit
[07:29] <rogpeppe> jam: i've found how that panic can happen, i think
[07:54] <TheMue> rogpeppe: you've once discussed https://bugs.launchpad.net/juju-core/+bug/1202163 with hazmat
[07:54] <_mup_> Bug #1202163: openstack provider should have config option to ignore invalid certs <papercut> <juju-core:Triaged by themue> <https://launchpad.net/bugs/1202163>
[07:54] <mgz> okay, nearly reboot for meeting time
[07:54] <mgz> TheMue: I'm probably the best person to ask about that bug
[07:54] <TheMue> rogpeppe: could you give me a hint where this change has to be done
[07:55] <TheMue> mgz: ok, so we'll continue after the meeting, thanks
[07:55] <TheMue> mgz: that you can reboot now ;)
[07:55] <rogpeppe> TheMue: it's probably something that needs a change in goose too
[07:56] <rogpeppe> TheMue: basically we need some way to tell goose that it should ignore unknown certs
[07:56] <rogpeppe> TheMue: within goose, it would need to set the TLS config on the https request that it makes to have InsecureSkipVerify=true
[07:57] <TheMue> rogpeppe: ah, this makes it more clear. already looked in our code as a first step, but not goose
[07:58] <rogpeppe> TheMue: cool
[08:01] <mgz> TheMue: posted a comment in the bug
[08:03] <TheMue> mgz: thx
[08:04] <thumper> davecheney: coming to the meeting?
[08:43]  * fwereade_ goes to get breakfast
[08:44] <jam> rogpeppe: you mention that you might have a solution for the TestManageStateServesAPI bug?
[08:44] <rogpeppe> jam: for the panic(no error) problem anyway, i think, yes
[08:45] <jam> fwereade_: what did you decide on: https://code.launchpad.net/~fwereade/juju-core/prepare-leave-scope/+merge/181065
[08:45] <jam> the thread went long, and sort of ended on "maybe this is or isn't correct"
[08:46] <rogpeppe> jam: i believe it's because the underlying State is being closed, which causes the state.Watcher to be stopped and return without an error
[08:46] <rogpeppe> s/state\.Watcher/watcher.Watcher/
[08:46] <jam> rogpeppe: so still just a race condition, right?
[08:46] <rogpeppe> jam: not really
[08:46] <jam> as in we expect the Watchers to be closed first
[08:46] <jam> but in this case the State got closed.
[08:47] <jam> rogpeppe: given it doesn't always fail it is clearly *some* sort of race condition :)
[08:47] <rogpeppe> jam: hmm, there's definitely some racing involved, yes
[08:48] <rogpeppe> jam: but i think it's wrong that the state watchers assume that because the underlying state has been closed they can panic
[08:48] <rogpeppe> jam: you're probably right that the watchers should probably be closed nicely before the state is closed
[08:49] <jam> rogpeppe: so I think the MustErr thing is because in production they want to expect that if they are shutting down, there is a reason for it.
[08:49] <jam> rogpeppe: But I find it very strange to panic when you *don't* have an error :)
[08:50] <jam> rogpeppe: there *might* be a case where we purposefully trigger something like that during upgrade to force the process to restart, but I'm not 100% sure how all that ties together.
[08:50] <rogpeppe> jam: the usual reason for MustErr is that if you're the code solely in control of a watcher, you *know* that it can't have been stopped by something else
[08:50] <rogpeppe> jam: so if it dies without an error there's something weird enough going on to warrant a panic
[08:50] <jam> fwereade_: thinking of upgrading. You had a comment about WatchAPIVersion (I think) being unhappy before we've gotten our env credentials set up. Is it just that you don't want that API call to return until we're ready to handle FindTools requests, or ?
[08:51] <rogpeppe> jam, fwereade_: FWIW i don't think it's a good idea to make that call block indefinitely until there's a valid environ config
[08:52] <rogpeppe> jam, fwereade_: because that could take forever and there's no way of interrupting that call once it's in progress
[08:53] <rogpeppe> fwereade_:
[08:53] <rogpeppe> jam, fwereade_: although having said that, i'm not sure i can think of a better alternative
[08:54] <jam> rogpeppe: well, avoiding panic will be a good first step towards figuring out what is going wrong. :)
[08:54] <rogpeppe> jam: indeed
[08:55] <jam> rogpeppe: because I think fwereade_ said jpds managed to get 'juju status' to run and either we still have the log-replay bug, *or* it was still failing on something.
[08:55] <jam> I was very surprised to see the err was a "schema.error_" which sounds like it is coming from somewhere else.
[08:56] <rogpeppe> jam: that panic should have been fixed by the recent ServerError change
[08:56] <rogpeppe> jam: that error comes when a schema doesn't match
[08:59] <fwereade_> jam, it's Tools in particular
[08:59] <fwereade_> jam, rogpeppe: I am relatively unbothered by blocking forever there
[08:59] <fwereade_> jam, rogpeppe: and using the environ in the first place is somewhat suboptimal
[09:00] <rogpeppe> fwereade_: yeah, i was just wondering about that - we're only using for agent-version, right?
[09:00] <rogpeppe> s/using/using it/
[09:00] <fwereade_> jam, rogpeppe: so at some point it will wither away regardless, in favour of a cache of simplestreams data
[09:00] <jam> fwereade_: as in, WatchAPIVersion runs, finds stuff, and then we call FindTools but that blocks until we have creds?
[09:00] <fwereade_> rogpeppe, nope
[09:00] <fwereade_> rogpeppe, it's FindExactTools that's the problem
[09:01] <fwereade_> rogpeppe, I'm 80% sure that we manage to extract agent-version, for the watcher, without creating an environ
[09:01] <fwereade_> jam, yeah
[09:01] <rogpeppe> fwereade_: oh, of course
[09:02] <rogpeppe> fwereade_: i think i agree that we can block forever - it doesn't mean we can't still respond the Upgrader.Kill, as long as we run the Tools call in a new goroutine
[09:02] <rogpeppe> s/respond the/respond to/
[09:03] <jam> fwereade_: well one option would be to add one more api which is "State.DesiredAgentVersion()" rather than having it always be Tools with the URL to get the new tools.
[09:03] <jam> fwereade_: because then it will go quiescent without having to search the provider.
[09:03] <fwereade_> rogpeppe, WaitForEnviron requires a done chan orsomething already iirc
[09:03] <jam> fwereade_: the change that worker/upgrader/upgrader.go did
[09:03] <jam> is that on startup for every agent it must go scan the bucket
[09:03] <fwereade_> jam, what's the utility there?
[09:03] <rogpeppe> fwereade_: but we're talking server side here, right?
[09:04] <jam> I believe we did it that way because it seemed silly to do 2 api requests
[09:04] <jam> but given that 1 is expensive (must read the provider buckets)
[09:04] <jam> and one is cheap (just from the db)
[09:04] <jam> maybe it is worth splitting up
[09:04] <fwereade_> jam, istm that it just complicates things without delivering any benefits?
[09:05] <jam> fwereade_: it very specifically moves us closer to how we used to do it, and would avoid this bug
[09:05] <jam> fwereade_: as the Upgrader would say "what version do you want me to be at?" and the response would match currentTools.version and we would go back to waiting on a change
[09:05] <jam> fwereade_: and we don't need provider creds to get that answer.
[09:06] <jam> rogpeppe: we got Unauthed access again: https://code.launchpad.net/~thumper/juju-core/container-address/+merge/182271
[09:07] <jam> mgz: it has already been approved, but if you could look over https://code.launchpad.net/~thumper/juju-core/container-address/+merge/182271 to see how it fits with your Addresses stuff.
[09:07] <rogpeppe> jam: that is *so weird*
[09:07] <fwereade_> jam, ah, ok... maybe that leads to a nicer server-side implementation too?
[09:07] <jam> rogpeppe: I agree, it is clearly a test that is an issue
[09:07] <jam> fwereade_: it splits the bits nicely
[09:07] <jam> fwereade_: and is probably really easy to implement
[09:08] <fwereade_> jam, ok, consider it blessed, thanks :)
[09:08] <jam> fwereade_: do we have a bug about failing during startup in 1.13.x ?
[09:11] <mgz> jam: is this the one I read through the other day, looking...
[09:11] <jam> fwereade_: also, this probably only happens when you are using a private bucket that requires creds.
[09:11] <jam> mgz: if you read thumper's MP you didn't respond to it
[09:12] <jam> fwereade_: it also means that we don't scan the provider every time *anything* in environconfig changes
[09:13] <jam> since we don't have smarts about just noticing api version changes yet
[09:16] <rogpeppe> jam: i'm not sure i see how DesiredAgentVersion would help
[09:16] <rogpeppe> jam: we'd still need to wait for the environment config to be valid, wouldn't we?
[09:17] <mgz> jam: right, didn't comment, still not completely sure on the implications (but as a hack, it seemed fine)
[09:22] <rogpeppe> afk
[09:35] <jam> rogpeppe: we write AgentVersion into the environment config during bootstrap. We don't yet have *provider* (eg EC2/MaaS/Openstack) credentials.
[09:35] <jam> rogpeppe: the existing Tools() api requires searching the provider.Storage for the binary that matches the desired version
[09:35] <jam> rogpeppe: because it returns the URL to download those tools.
[09:35] <jam> which is a problem during bootstrap because we don't actually have the credentials to search Storage yet.
[09:35] <jam> well, "first boot"
[09:40] <rogpeppe> jam: ah yes, of course
[09:41] <rogpeppe> jam: i think that separating the API calls makes a lot of sense actually
[09:41] <rogpeppe> jam: then the relationship between the version watcher and the API call we're making is obvious
[10:19] <fwereade_> jam, I'm feeling somewhat nauseous and it isn't going away, I'm going to lie down for a while
[10:20] <jam> fwereade_: shame, I'm just about to propose the fix. Feel better soon.
[10:20] <rogpeppe> fwereade_: hope it *does* go away
[10:20] <fwereade_> jam, if I don't come back for the meetings please make my apologies
[10:20] <fwereade_> jam, I will surely at least manage some reviewing later today though
[10:21] <fwereade_> rogpeppe, cheers )
[10:22] <jam> davecheney: if you are around, didn't you have a bug about seeing listing-the-storage-bucket over and over in the logs?
[10:22] <jam> I think my fix will also address that
[10:25] <jam> rogpeppe: if you have time https://codereview.appspot.com/13380043/
[10:33]  * TheMue => lunchtime
[10:49] <rogpeppe> jam: reviewed
[10:51] <jam> rogpeppe: so I was following the example of "tools.Tools" which returns a pointer rather than a struct.
[10:51] <jam> Is there a reason they should be different?
[10:52] <rogpeppe> jam: just convention
[10:52] <rogpeppe> jam: we always pass around *tools.Tools
[10:52] <rogpeppe> jam: and version.Number
[10:53] <rogpeppe> jam: if we want to change it, we should do it consistently across the code base
[10:53] <jam> rogpeppe: what about the API itself then? I think it was using *tools so that it just puts nil there when it say, has an error.
[10:53] <jam> it seems a little strange to serialize a bunch of 0's onto the wire when you have an error.
[10:54] <rogpeppe> jam: omitempty might possibly work
[10:54] <rogpeppe> jam: although it may not work on structs, come to think of it
[10:55] <rogpeppe> jam: i don't mind if the serialisation struct uses a pointer, if that's what we need to lose it from the RPC resulyt
[10:56] <mgz> what is "empty" in the context of a struct... can you test for nilledness easily with go?
[10:56] <mgz> zeroedness actually I guess
[10:56] <jam> mgz: foo == Foo{}
[10:56] <rogpeppe> mgz: yeah
[10:56] <rogpeppe> mgz: but it's not that easy to check for using reflection, so it's quite probably it doesn't do that
[10:57] <rogpeppe> probable even
[10:57] <mgz> that makes sense
[10:58] <rogpeppe> mgz: yeah, it doesn't do that
[11:06] <mgz> jam: any ideas why sync-tools --source is trying to list tools before doing anything?

[11:06] <mgz> on canonistack, so may involve simplestreams...
[11:06] <jam> mgz: as in why it would be listing ~/go/bin or listing something upsteram?
[11:06] <jam> it normally lists the target tools
[11:06] <jam> so it can find what tools need to be copied
[11:06] <mgz> it's trying to list a swift container. it should just splat up the local stuff, no?
[11:07] <jam> mgz: not when you already have stuff
[11:07] <jam> it is "put stuff I don't have"
[11:07] <jam> not "overwrite everything"
[11:07] <jam> mgz: that way if I get interupted after copying 3 things, I don't start from scratch again.
[11:07] <mgz> but... this is then a catch-22. have no tools, can't upload my local tools, because I have not tools?
[11:07] <jam> not that it should fail if it can't find any target tools
[11:07] <jam> mgz: I think you are misreading it. I think it is failing because it can't find any *source* tools to copy
[11:08] <mgz> jam: that dir certainly contains jujud
[11:09] <mgz> all: will miss standup, things to mention, filed a selection of goose bugs about issues related to rackspace
[11:09] <mgz> and er... I'm being prodded
[11:11] <jam> mgz: tools are usually a foo-series-arch-number.tar.gz sort of thing
[11:16] <arosales> jam, https://bugs.launchpad.net/juju-core/+bug/1216768
[11:16] <_mup_> Bug #1216768: Azure provider: Authentication error when using public tools <juju-core:Fix Committed by axwalk> <https://launchpad.net/bugs/1216768>
[11:32] <jam> rogpeppe: https://plus.google.com/hangouts/_/f497381ca4d154890227b3b35a85a985b894b471 standup?
[11:49] <arosales> jam, https://bugs.launchpad.net/juju-core/+bug/1218329 is the other Azure bug needed.
[11:49] <_mup_> Bug #1218329: Update default environment.yaml for Azure to use Precise for default-series <juju-core:New> <https://launchpad.net/bugs/1218329>
[11:50] <arosales> jam what method would you suggest to test trunk with a public tools? sync-tools to upload the current trunk tools to a public bucket?
[11:50] <jam> arosales: you can create a different "public" bucket and sync-tools into it, I think.
[11:51] <jam> and then configure you environment to treat that as the actual public bucket.
[11:52] <arosales> jam, ok I'll give that a try to day thanks.
[11:52] <jam> arosales: for bug #1218329 you've confirmed we can switch to precise? because it is pretty trivial to change that one line.
[11:52] <_mup_> Bug #1218329: Update default environment.yaml for Azure to use Precise for default-series <juju-core:New> <https://launchpad.net/bugs/1218329>
[11:53] <arosales> jam, the other point I need to confirm is the image-stream.
[11:53] <arosales> I need to confirm if we are calling the latest precise images with the fix as "released" or "daily" in simple streams and in Azure publication.
[11:53] <arosales> jam, I also added a comment with that.
[12:07] <jam> rogpeppe: I think I responded to all of your requests: https://codereview.appspot.com/13380043/
[12:08] <jam> arosales: I won't be around tomorrow to make sure azure things are addressed before the cut of the release (usually happens on the weeked). You can probably poke some of the people who are in Europe during your morning (mgz, fwereade_, allenap come to mind)
[12:09] <jam> hey mramm, we didn't think we'd see you today
[12:09] <mramm> jam: I had the overnight flight to london
[12:09] <mramm> just checking in at the hotel
[12:09] <arosales> jam, I'll sync with davecheney and fwereade_  on it if a release is targetted this week
[12:12] <jam> arosales: I fully expect at least 1.13.3 to be out this weekend. And we'd like to release it, test it, and possible directly upgrade it to 1.14
[12:14] <arosales> jam, ok, and if a stable release causes a lot of pain and you just go with devel just let me know, for azure documentation set up purposes.
[12:18] <rogpeppe> jam: reviewed
[12:40] <TheMue> jam: due to doc sprint tomorrow and now having discovered that it's a deeper goose change too i would like to see lp:1202163 reassigned to one of the goose team. what do you say?
[12:42] <rogpeppe> jam, fwereade_: state.RelationUnitsWatcher doesn't seem to have any tests at all. do you know what's going on there?
[12:50] <rogpeppe> hmm, looks like it was deleted in https://codereview.appspot.com/7198051/
[12:54] <rogpeppe> https://bugs.launchpad.net/juju-core/+bug/1218362
[12:54] <_mup_> Bug #1218362: state.RelationUnitWatcher is not tested <tech-debt> <juju-core:New> <https://launchpad.net/bugs/1218362>
[13:36]  * rogpeppe just ran the coverage test tool for the first time
[13:36] <rogpeppe> it works well
[13:36] <natefinch> nice
[13:36] <rogpeppe> 85% coverage of the state package
[13:36] <natefinch> hey, 85% is really good
[13:36] <rogpeppe> natefinch: it would be 86.9% if we actually tested RelationUnitWatcher
[13:37] <rogpeppe> aw, "cannot use test profile flag with multiple packages"
[13:37] <natefinch> rogpeppe: heh. I don't put a ton of stock in coverage, since covered doesn't necessarily mean tested... but not covered is definitely not tested
[13:37] <rogpeppe> natefinch: that's my thought
[13:38] <rogpeppe> natefinch: i was wondering about things that would have made it obvious that we'd lost test coverage
[13:38] <natefinch> rogpeppe: so, only 15% definitely not tested, and 85% that is at least exercised, so that's pretty great
[13:38] <rogpeppe> natefinch: (as happened in this CL in january: https://codereview.appspot.com/7198051/ )
[13:39] <natefinch> rogpeppe: yeah, detecting when we lose coverage is a good idea
[14:01] <rogpeppe> jam: if you're still around, this fixes the watcher panic: https://codereview.appspot.com/13386044/
[14:01] <rogpeppe> fwereade_, natefinch, mgz: reviews appreciated
[14:02]  * rogpeppe goes for some lunch
[14:03] <natefinch> rogpeppe: I'll take a look
[15:21] <arosales> any juju core folks around to attend vUDS session: http://summit.ubuntu.com/uds-1308/meeting/21899/servercloud-s-juju-new-user-ux/
[15:22] <arosales> rogpeppe,  ^
[15:22]  * arosales was going to bother fwereade but I don't see him online atm
[15:23] <rogpeppe> arosales: he's sick atm, i think
[15:23] <arosales> ah, sorry to hear
[15:23] <rogpeppe> arosales: what time zone is that time in?
[15:23] <arosales> rogpeppe, utc
[15:23] <arosales> so roughly in half hour
[15:23] <rogpeppe> arosales: ok, i'll be there
[15:24] <arosales> rogpeppe, much appreciated, thank you
[15:28] <rogpeppe> arosales: do i have to register as attending to join?
[15:28] <arosales> rogpeppe, I don't think so
[15:28] <arosales> rogpeppe, I'll post the hangout url in the channel here in a bit, and we can take it from there.
[15:29] <rogpeppe> arosales: ah, it's a hangout - i was wondering where the video was
[15:29] <rogpeppe> arosales: thanks
[15:30] <arosales> rogpeppe, yup live hangout, but I haven't started it just yet
[15:30] <arosales> still have about 20 minutes
[15:47] <rogpeppe> natefinch: i've addressed your concerns, i hope: https://codereview.appspot.com/13386044
[15:48] <rogpeppe> natefinch: or replied, at any rate :-)
[15:57] <natefinch> rogpeppe: cool, I'll take a look.  Would like someone more familiar with the problem to give it the lgtm if possible, though
[15:58] <rogpeppe> natefinch: yeah, i guess i'll have to wait until tomorrow unless mgz or jam are around
[16:00] <mgz> the watcher change makes some sense to me, but I'm not sure we have more qualified than rog on the problem :)
[16:01] <mgz> what's the implication of a one-value return from a channel, rather than the `, ok` form?
[16:09] <natefinch> rogpeppe: it sorta bugs me that Watcher is an interface without a Watch() method :p
[16:09] <rogpeppe> please think of a better name!
[16:09] <rogpeppe> natefinch: ^
[16:12] <natefinch> rogpeppe: I would almost call it Stopper or Ender... it's the  Changes() method that really makes a watcher a watcher
[16:14] <rogpeppe> natefinch: i originally thought about Worker
[16:14] <natefinch> rogpeppe: yeah, I was thinking something like that originally
[16:16] <natefinch> arosales: you should email the team ahead of time so we can plan on being around for these meetings :)
[16:17] <arosales> natefinch, you guys didn't know about vUDS?
[16:18] <arosales> natefinch, but noted I don't think I included juju-core folks in my uds reminders
[16:18] <natefinch> arosales: reminders are much appreciated :) Also, I'm new, so I might not know things I'm supposed to know :)
[16:18] <arosales> natefinch, for sure you get a pass :-)
[16:19] <arosales> but these other veterans  . . . .
[16:19] <arosales> natefinch, but I agree with you on reminders. I'll note for next time bother the juju-core folks :-)
[17:39] <weblife> Juan Negron in here?
[18:27]  * rogpeppe has reached eod
[18:27] <rogpeppe> g'night all
[21:31] <thumper> morning folks
[23:18]  * thumper is at that WTF moment debugging
[23:24]  * thumper grunts...
[23:37] <thumper> found the source of my confusion
[23:37] <thumper> simplified to this:
[23:37] <thumper>     c.Assert(obtained, gc.DeepEquals, expected)
[23:37] <thumper> ... obtained []uint8 = []byte{}
[23:37] <thumper> ... expected []uint8 = []byte{}
[23:39] <thumper> bigjools: the magic of gocheck  :)
[23:39]  * thumper fixes
[23:40] <bigjools> thumper: wtf
[23:40] <thumper> bigjools: I know, right?
[23:40] <thumper> had me scratching my head for quite a while
[23:40] <bigjools> what is it hiding?
[23:40] <thumper> it is outputting something incorrectly
[23:40] <thumper> here are the two lines prior
[23:41] <thumper> 	obtained := []byte{}
[23:41] <thumper> 	var expected []byte
[23:41] <thumper> obtained is an empty slice
[23:41] <thumper> expected is a nil slice
[23:45] <davecheney> thumper: we need a SliceEquals checker
[23:45] <thumper> what is shown here is just a part of the problem
[23:45] <thumper> I'm checking an entire structure
[23:45] <thumper> using deep equals
[23:45] <thumper> the slice is just one part
[23:47] <thumper> a key part of the problem is that it isn't showing []byte{nil}
[23:47] <thumper> which is what *should* be shown for a nil slicke
[23:47] <thumper> slice
[23:47] <thumper> that would have made the problem completely obvious
[23:47] <thumper> instead of hiding it
[23:48] <thumper> oh...
[23:48] <thumper> check this out:
[23:48] <thumper> http://play.golang.org/p/LRDkBszMNa
[23:48] <thumper> vs http://play.golang.org/p/iO2wpSUxeO
[23:49] <thumper> a nil string slice is output as []string(nil), but a nil byte slice is shown as []byte{}
[23:49] <thumper> however, a nil byte slice does not equal an empty byte slice
[23:50] <thumper> davecheney: go bug?
[23:53] <davecheney> nope
[23:54]  * bigjools boggles
[23:56] <thumper> davecheney: why?
[23:56] <davecheney> otp


[00:52] <axw> wallyworld: I think you misunderstood me about the API
[00:52] <axw> I just meant to combine them in the CLI code
[00:52] <wallyworld> ah, damn
[00:52] <axw> i.e. create an interface that you can satisfy by embedding both the facades into one struct
[00:53] <wallyworld> oh well, what's there is ok i think?
[00:53] <wallyworld> we use that same pattern elsewhere too
[00:53] <axw> wallyworld: I think it's fine
[00:53] <axw> wallyworld: ModelManager makes more sense for this command anyway I think
[00:53] <wallyworld> yeah, agree
[01:23] <rick_h_> wallyworld: is the model destroy blocking something communicated to other teams?
[01:23] <rick_h_> wallyworld: coukd tgat imoact things like the gui?
[01:23] <wallyworld> rick_h_: it was a stakeholder bug from OIL I think. I am not sure who else knows
[01:24] <wallyworld> i am not sure about GUI - I'll send an emil
[01:24] <wallyworld> rick_h_: but the blocking is ony CLI
[01:24] <rick_h_> wallyworld: right, i'm +1000 on the change, but want to make sure we chat with ither clients
[01:24] <rick_h_> wallyworld: ah ok
[01:24] <wallyworld> the api behaves the same
[01:24] <rick_h_> wallyworld: nvm then
[01:25] <wallyworld> fair enough question to asl
[01:25] <wallyworld> *ask
[01:48] <menn0> thumper: i've fixed the racy pingerSuite tests: https://bugs.launchpad.net/juju/+bug/1625214
[01:48] <mup> Bug #1625214: pingerSuite.TestAgentConnectionsShutDownWhenAPIServerDies write: broken pipe <ci> <intermittent-failure> <regression> <unit-tests> <juju:In Progress by menno.smits> <https://launchpad.net/bugs/1625214>
[01:48] <menn0> thumper: what I meant to paste: https://github.com/juju/juju/pull/6410/commits/f9ee5dba43e0fa032fe67d14d1c89cdc1c1e1a55
[01:49] <thumper> looking
[01:52] <thumper> approve
[01:52] <menn0> thumper: cheers
[02:01] <menn0> wallyworld: https://github.com/juju/juju/pull/6411 please
[02:02] <wallyworld> menn0: sure, just finishing another
[02:02] <menn0> wallyworld: ignore the first commit. that one is merging as part of another PR now.
[02:02] <wallyworld> ok
[02:19] <wallyworld> menn0: could you take a look at this one by christian? i think your +1 would be valuable. I've already had a look https://github.com/juju/juju/pull/6408
[02:20] <wallyworld> also, your 6411 pr has conflicts
[02:20] <menn0> wallyworld: will do. he emailed me about it too.
[02:20] <wallyworld> ta
[02:21] <menn0> wallyworld: conflicts fixed
[02:21] <wallyworld> ta
[02:53] <axw> menn0: are you investigating that cert issue?
[02:54] <menn0> axw: I was about to take a short break and then stuck into it
[02:54] <menn0> axw: I still have a controller up where that's happened
[02:54] <menn0> axw: have you seen it too?
[02:54] <axw> menn0: ok. would it be helpful if I looked in parallel?
[02:54] <axw> menn0: no, but it didn't sound hard to repro
[02:55] <menn0> axw: could be. if you could take a look too that would be great.
[02:55] <axw> menn0: ok. I'll let you know if I find anything
[02:55] <menn0> axw: here's what the failure looks like: http://paste.ubuntu.com/23301342/
[02:56] <axw> okey dokey
[02:56] <axw> menn0: and all you did was bounce the controlelr agent?
[02:57] <menn0> axw: I stopped the lxd machine and started it again
[02:57] <axw> ah yes
[02:57] <axw> ok
[02:57] <menn0> the controller instance
[02:57]  * menn0 will be back in 15 mins
[03:17] <axw> wallyworld: what did you want me to test with add-model? something like this? bootstrap rc2, upgrade to my branch, then add-model foo && add-model bar ap-southeast-2
[03:17] <wallyworld> axw: yeah, check that the endpoint fixing is done for addmodel also
[03:18] <wallyworld> axw: or even first confirm that it is broken without your fix
[03:18] <wallyworld> so we can be sure that the fix is relvant nd valid
[03:18] <axw> wallyworld: it would be broken if you tried to upgrade from rc2
[03:18] <axw> wallyworld: but if you added a model to a running rc2 without upgrading, it should work. anyway, I'll try what I described
[03:19] <wallyworld> ta, i think we need to be 10000% sure
[03:40] <axw> menn0: I strongly suspect -> https://github.com/juju/juju/commit/0d64508f8005ea84c141ed392389fbaef0e70b30
[03:40] <axw> menn0: in particular, "info.Cert = existing.Cert" without also setting Key
[03:51] <blahdeblah> Hi everyone, which ppa should I be using to get the RC version of juju 2.0 so I can try https://docs.google.com/document/d/1yT5pvS38g9Z9SviI9NPmhIZrsO8drwHDPNWAMSmCedE/edit# ?
[03:52] <menn0> axw: looks like there was already a ticket for this issue: https://bugs.launchpad.net/juju/+bug/1631145
[03:52] <mup> Bug #1631145: rc2 upgrade to rc3 failed with cannot start api server worker: cannot set initial certificate: cannot create new TLS certificate: crypto/tls: private key does not match public key <juju:New> <https://launchpad.net/bugs/1631145>
[03:52] <menn0> axw: i've just bumped the priority etc
[03:54] <axw> okey dokey
[03:54] <menn0> axw: any luck replicating?
[03:55] <axw> menn0: not yet
[03:55] <axw> blahdeblah: according to release email,  ppa:juju/devel
[03:55] <blahdeblah> axw: cool - thanks
[03:58] <wallyworld> axw: with maas, do you know what happens if you attempt to acquire a node without specifying an architecture? does it just pick some arbitary machine so long as any other constraints match?
[03:58] <axw> menn0: hrm. I am seeing "getsockopt...connection refused" without the other error though
[03:58] <axw> and status isn't working now
[03:58] <axw> wallyworld: I don't recall, sorry
[03:58] <menn0> axw: that's what you'll see at the client
[03:58] <menn0> axw: what about in the controller's logs?
[03:59] <wallyworld> maybe menn0 recalls?
[03:59] <axw> menn0: this is in the controller machine log
[03:59] <menn0> axw: hmm ok
[03:59] <axw> menn0: it's failing to connect to mongo...
[03:59] <menn0> axw: possibily a different problem?
[03:59] <axw> yes, possibly
[04:00] <anastasiamac> wallyworld: r:maas arch.. yes, i beleve so.. at lest this s what i ve seen
[04:00] <menn0> axw: not ideal
[04:00] <axw> menn0: although... do we use the same cert/key for mongo?
[04:00] <wallyworld> menn0: ta, ok
[04:00] <menn0> wallyworld: sorry, not sure about maas
[04:00] <wallyworld> np, i'll ask the maas qguys
[04:00] <wallyworld> guys
[04:01] <menn0> axw: maybe we do use the same cert/key... worth checking
[04:03] <axw> menn0: it does appear to be the case
[04:04] <axw> when we update the cert, we write out to server.pem
[04:04] <menn0> axw: ok, so probably a different manifestation of the same issue
[04:04] <axw> yup
[04:04] <menn0> axw: i'm tracing through where the cert and key that apiserver.NewServer is given come from
[04:06] <menn0> axw: ok so that's out of StateServingInfo in the agent config
[04:06] <menn0> axw: perhaps one is being updated without the other?
[04:14] <menn0> axw: this could be it: https://github.com/juju/juju/blob/master/cmd/jujud/agent/machine/servinginfo_setter.go#L67-L74
[04:15] <menn0> axw: should existing.PrivateKey get copied too?
[04:19] <menn0> axw: I'd also be a lot happier if most of what the stateservinginfo_setter did happened /inside/ the ChangeConfig
[04:19] <menn0> axw: same for the method which does cert updates for the apiserver
[04:19] <menn0> looks racy otherwise
[04:24] <menn0> axw: the more I look, the more I think this is the bug
[04:24]  * menn0 fixes
[04:25] <axw> menn0: sorry, I just realised you lost connection after I pasted the commit before :/
[04:25] <menn0> axw: did you come to the same conclusion?
[04:25]  * menn0 has been having wifi problems
[04:25] <axw> [11:40:32] <axw> menn0: I strongly suspect -> https://github.com/juju/juju/commit/0d64508f8005ea84c141ed392389fbaef0e70b30
[04:25] <axw> [11:40:57] <axw> menn0: in particular, "info.Cert = existing.Cert" without also setting Key
[04:25] <menn0> axw: ha! did see that. we independently found the same thing
[04:26] <menn0> axw: s/did/didn't/
[04:27] <menn0> axw: do you also agree that much of the logic in stateservinginfo_setter should be /inside/ the ChangeConfig func?
[04:28] <menn0> axw: same for MachineAgent.upgradeCertificateDNSNames
[04:28] <axw> menn0: yes, I think so
[04:28] <axw> anything looking at current config
[04:29] <menn0> axw: ok, let's fix it.
[04:29] <menn0> axw: have you already started?
[04:29] <axw> menn0: nope, eating lunch atm
[04:30] <wallyworld> blahdeblah: i so want to mark your bug as invalid. using firefox as default browser should be invalid :-)
[04:30] <menn0> axw: ok, i'll start on servinginfo_setter.go.
[04:30] <menn0> axw: i'm going to have to stop soon but will be back on later this evening
[04:32] <blahdeblah> wallyworld: Feel like taking the gloves off today, eh? :-)
[04:32] <axw> menn0: ok. let me know where you leave off, I can pick it up
[04:32] <menn0> axw: sounds good
[04:32] <wallyworld> blahdeblah: i used to run firefox until it started to grind everything to a halt with many tabs open and other performance things
[04:33] <blahdeblah> wallyworld: It has never done that for me.  I've heard many claims of such, but never seen it personally, despite having 4 windows with 100-odd tabs open right now.  I thank NoScript for that.
[04:34] <wallyworld> blahdeblah: yeah, that may well be it. sadly lots of sites require js, including lp. well lp doesn't need it but uscks without it
[04:34] <blahdeblah> I use js on lots of sites; I just don't let it do that by default
[04:35] <wallyworld> i find it too hard to whitelist everything. well maybe am too lazy
[04:35] <blahdeblah> anyhew, my bug is valid, and I'll fight you for it! :-P
[04:35] <wallyworld> adblock works a treat though
[04:35] <wallyworld> oh i know it is valid
[04:35] <wallyworld> was just stirring
[04:36] <blahdeblah> I see your stirring, and raise you lp:1587644, which got me out of bed at least once this weekend
[04:37] <anastasiamac> menn0: wallyworld: considering u've re-viewed the fix for bug 1625774
[04:37] <mup> Bug #1625774: memory leak after repeated model creation/destruction <ateam> <eda> <oil> <oil-2.0> <uosci> <juju:In Progress by 2-xtian> <https://launchpad.net/bugs/1625774>
[04:38] <anastasiamac> menn0: wallyworld: is it possible/wanted/needed on 1.25 to solve some of the memory leakage there too?
[04:39] <wallyworld> not sure, depends on when state pool was introduced. not sure if it was for multi model or not
[04:39] <wallyworld> would have to dig in and look
[04:39] <menn0> anastasiamac: it could be applied there but I don't think multi-model was supported without a feature flag was it? the fix would have minimal utility if that's the case.
[04:45] <wallyworld> yes, multi model in 1.25 is ff
[04:46] <anastasiamac> wallyworld: even server-side stuff? we usually ff at cli level...
[04:46] <wallyworld> not for multi model
[04:46] <wallyworld> IIRC
[04:47] <menn0> yeah, I'm pretty sure the feature flag was applied throughout the server too
[04:52] <menn0> axw: I have the ssi setter fixes done, pushing now
[04:52] <axw> cool
[04:52] <menn0> axw: can I leave the MachineAgent.upgradeCertificateDNSNames fix to you?
[04:53] <axw> menn0: yep sure
[04:54] <menn0> axw: https://github.com/juju/juju/pull/6413
[04:55] <axw> menn0: I think the only thing to do there is to not use "si.Cert", but get fresh value inside ChangeConfig?
[04:55] <axw> ta
[04:55] <mup> Bug #1630728 changed: remove user  needs better message that user is made inactive <usability> <juju:Triaged> <https://launchpad.net/bugs/1630728>
[04:57] <menn0> axw: shouldn't cert.NewDefaultServer be called inside the ChangeConfig though?
[04:57] <axw> menn0: yes, I mean everything from "Parse the current certificate to get the current dns names." down should be inside
[04:58] <menn0> axw: +1 that's what I was thinking
[04:58] <menn0> should be easy
[04:58] <menn0> axw: just trying to QA this PR before I get called to help out with other stuff
[04:58] <axw> menn0: except maybe the mongo bit. might want to do that after writing agent conf
[04:58] <axw> menn0: okey dokey, thanks
[04:59] <menn0> axw: maybe... it might be ok to do it in the changeconfig though
[04:59] <menn0> axw: actually, i'm out of time. if you could review that PR, I'll check back in later and do the QA before merging
[05:00] <axw> menn0: approved
[05:01] <axw> menn0: with one late comment
[06:30] <menn0> axw: I made that change and QAed. Merging now.
[06:33] <axw> menn0: thanks
[06:33] <axw> menn0: I'm just QAing my change now
[06:33] <menn0> axw: cool. i can review when it's up. i've got to do dishes etc but will check back in later.
[06:33] <axw> menn0: https://github.com/juju/juju/compare/master...axw:lp1631145-upgradecertificatednsnames?expand=1 if you want to take quick look
[06:33] <axw> ok, no woprries
[07:14] <axw> wallyworld: got anything smallish you'd like help?
[07:16] <wallyworld> axw: not off hand. I am finishing a partial list-clouds fix pending input from rick et al. i think there's one remaining bug on alexis' list to do with agent restart but not sure how big it is
[07:16] <axw> wallyworld: sounded big, I'll take a look
[07:17] <wallyworld> yeah, it did at first look
[07:52] <wallyworld> axw: is your cert fix just because we need to recover from a borked cert cause by rc3?
[07:52] <axw> wallyworld: yes. and also we should be doing stuff inside ConfigChanged in general. doesn't matter in this case because the function is only called at startup
[07:53] <axw> but good to keep it clean, to avoid copy&paste errors
[07:53] <wallyworld> +1
[07:53] <wallyworld> lgtm
[08:05] <hoenir> could someone review my PR ? https://github.com/juju/juju/pull/6414
[08:06] <hoenir> also note that this PR will be the foundations of a new feature I wish to unlock on juju, enabling manual provisioning for windows machines.
[08:07] <hoenir> foundation*
[08:11] <mup> Bug #1420996 opened: Default secgroup reset periodically to allow <IP_ADDRESS>/0 for 22, 17070, 37017 <canonical-is> <juju-core:New> <https://launchpad.net/bugs/1420996>
[08:58] <dooferlad> mgz: is the github jujubot happy? It seems to be ignoring my $$merge$$ on https://github.com/juju/juju/pull/6406
[09:20] <babbageclunk> wallyworld: ping?
[09:20] <wallyworld> yo
[09:20] <babbageclunk> wallyworld: I think I agree with you about Release vs Put - do you think menn0 will sulk if I change it?
[09:21] <wallyworld> babbageclunk: he can sulk, i'll get the popcorn ready :-D
[09:21] <babbageclunk> :)
[09:21] <wallyworld> babbageclunk: or even Done()
[09:25] <babbageclunk> wallyworld: Done isn't verby enough for my taste. Get/Done doesn't seem right.
[09:25] <wallyworld> fair point, i was trying to find an alternative to Release() to aleviate objections :-)
[09:27] <babbageclunk> always the peacemaker
[09:27] <wallyworld> oh, i have been known to enjoy poking the hornet's nest :-D
[09:42] <babbageclunk> wallyworld: do you think I should rip the non-strict model uuid checking out of validateModelUUID?
[09:43] <babbageclunk> wallyworld: There are still a few places that use it.
[09:43] <wallyworld> babbageclunk: i *think* so - maybe in a followup after checking with tim/menno. i am pretty sure it was just to support really old clients
[09:44] <babbageclunk> wallyworld: Ok - I'll do it as a separate change.
[09:44] <wallyworld> sgtm
[10:02] <mup> Bug #1631899 opened: juju show-controller --show-password does not show the password <juju-core:New> <https://launchpad.net/bugs/1631899>
[10:21] <babbageclunk> wallyworld: Want to take another look at that PR, or are you ok for me to merge it?
[10:22] <wallyworld> babbageclunk: i can take a quick look, menno was happy with it
[10:25] <babbageclunk> wallyworld: cool thanks - I think I've gone through all of your comments.
[10:28] <wallyworld> babbageclunk: i think it looks good to go. thanks for doing the fix; was a challenege to get all the bit lined up so you did well
[10:28] <babbageclunk> wallyworld: cheers!
[11:02] <beisner> hi all, last week we started seeing sha256 mismatches when units try to download the charm from the controller (1.25.6).  it's prevalent in openstack ci.  ie. shutting down: ModeInstalling ... failed to download charm ... expected sha256 FOO, got BAR
[11:02] <beisner> is there a known issue or are we special? :)   more detail @ http://pastebin.ubuntu.com/23302619/
[11:11] <mup> Bug #1631899 changed: juju show-controller --show-password does not show the password <juju-core:New> <https://launchpad.net/bugs/1631899>
[11:23] <mup> Bug #1631899 opened: juju show-controller --show-password does not show the password <juju-core:New> <https://launchpad.net/bugs/1631899>
[11:26] <mup> Bug #1631899 changed: juju show-controller --show-password does not show the password <juju-core:New> <https://launchpad.net/bugs/1631899>
[11:26] <mup> Bug #1541482 opened: unable to download local: charm due to hash mismatch in multi-model deployment <2.0-count> <juju-release-support> <juju:Fix Released by menno.smits> <juju-core:New> <https://launchpad.net/bugs/1541482>
[12:17] <mup> Bug #1629951 changed: cannot specify subnet to  create controller in on bootstrap <juju:Triaged> <https://launchpad.net/bugs/1629951>
[12:17] <mup> Bug #1630029 changed: models should inherit vpc-id from controller  <juju:Triaged> <https://launchpad.net/bugs/1630029>
[13:00] <dooferlad> rick_h_: 1:1 today?
[13:01] <rick_h_> dooferlad: /me checks thought he was in it
[13:01] <rick_h_> dooferlad: oh hmm, stuck at "requesting to join the video call"
[14:00] <rick_h_> katco``: dimitern voidspace mgz natefinch ping for standup
[14:00] <mgz> omw
[14:00] <dimitern> omw
[14:00] <rick_h_> oh right, US away so ignore me katco`` and nate
[14:02] <voidspace> rick_h_: omw
[14:49] <frobware> jamespage: do you often run into this issue: https://bugs.launchpad.net/juju/+bug/1600546
[14:49] <mup> Bug #1600546: lxd subnet setup by juju will interfere with openstack instance traffic <2.0> <network> <sts> <juju:Triaged by rharding> <nova-compute (Juju Charms Collection):New> <https://launchpad.net/bugs/1600546>
[14:51] <voidspace> rick_h_: ping
[14:51] <rick_h_> voidspace: pong
[15:21] <rick_h_> dooferlad: ping
[15:21] <dooferlad> rick_h_: hello
[15:21] <rick_h_> dooferlad: have more time to chat?
[15:22] <dooferlad> rick_h_: yes
[15:22] <rick_h_> dooferlad: k, meet you back in the 1-1 room
[15:32] <voidspace> rick_h_: 30 seconds for a bikeshed needed if you have it
[15:34] <voidspace> rick_h_: last comment on bug 1602192 is the output I've implemented
[15:34] <mup> Bug #1602192: when starting many LXD containers, they start failing to boot with "Too many open files" <lxd> <juju:In Progress by rharding> <lxd (Ubuntu):Confirmed> <https://launchpad.net/bugs/1602192>
[15:38] <rick_h_> voidspace: otp
[15:38] <rick_h_> voidspace: will look in a sec
[15:38] <voidspace> rick_h_: ok, np - I'll PR it and the reviewer can bikeshed it
[16:00] <jamespage> frobware, I've never run into that issue
[16:00] <frobware> jamespage: thanks. just trying to understand the severity and whether we address it "now".
[16:00] <jamespage> but apparently trent has hit it alot
[16:01] <frobware> jamespage: any feeling whether this is just happening for just the training setup?
[16:01] <jamespage> frobware, I'm not quite close enough to know the answer to that
[16:01] <jamespage> sorry
[16:02] <frobware> rick_h_: ^^ fyi
[16:02] <voidspace> dimitern: if you have any ideas on how to test this, I'm all ears
[16:02] <voidspace> dimitern: https://github.com/juju/juju/pull/6419
[16:02] <dimitern> voidspace: looking
[16:02] <voidspace> dimitern: except providing something that stubs out PrintLn.
[16:03] <voidspace> dimitern: the important bit is in provider/common/bootstrap.go
[16:04] <dimitern> voidspace: how about adding an interface with BootstrapMessage() method and then check in common/bootstrap if it's implemented by the Environ ?
[16:04] <dimitern> voidspace: then you could test it with the dummy provider, but no need to change all providers?
[16:05] <voidspace> dimitern: right, but what does that enable me to test?
[16:05] <voidspace> dimitern: I could then pass in a fake environ with a custom message, but what do I test?
[16:05] <voidspace> dimitern: unless I replace the call to fmt.Println with something that can be mocked - which doesn't seem very useful
[16:05] <voidspace> dimitern: I'm kind of arguing that as this is a ui change it needn't be tested
[16:06] <voidspace> dimitern: maybe I can look in featuretests to see if something like this is covered
[16:06] <dimitern> voidspace: only that common.Bootstrap() calls the optional method, when it's there, and checking the ctx.Stdout() to ensure it's there?
[16:06] <rick_h_> voidspace: wfm for now, thank you
[16:06] <voidspace> rick_h_: cool, thanks
[16:06] <rick_h_> frobware: yea, so I'm -1 on that being the biggest issue atm
[16:06]  * rick_h_ goes to get lunchables
[16:06] <voidspace> dimitern: in which case I can test that *already*
[16:07] <voidspace> dimitern: and put a non-empty string in dummy provider to test it
[16:07] <voidspace> dimitern: let me look - thanks
[16:07] <voidspace> dimitern: if we're calling fmt.Println will that be in ctx.Stdout ?
[16:07] <dimitern> voidspace: alternatively, you could go with BootstrapContext.Infof() only called in provider/lxd ?
[16:08] <dimitern> voidspace: since it's very specific and not needed everywhere
[16:09] <dimitern> voidspace: check e.g. PrepareForBootstrap in provider/ec2 and the code around validateBootstrapVPC()
[16:10] <dimitern> voidspace: (bootstrap)ctx.Infof() is already used for similar messages during bootstrap in some providers
[16:10] <voidspace> kk
[16:33] <voidspace> dimitern: using Stderr on the context I can test that the message is output
[16:34] <dimitern> voidspace: \o/ nice!
[17:55] <rogpeppe1> katco``: i've made a bunch of comments and changes in response to your review of https://github.com/juju/juju/pull/6407. PTAL.
[17:55] <rick_h_> rogpeppe1: she's on US holiday today
[17:56] <rick_h_> rogpeppe1: and with the EU folks EOD'ing will have to catch tomorrow it looks like sorry for the delay
[18:18] <arosales> If any folks are have access to s390 we are seeing https://bugs.launchpad.net/juju/+bug/1632030
[18:18] <mup> Bug #1632030: juju-db fails to start -- WiredTiger reports Input/output error <juju> <juju-db> <mongodb> <s390x> <juju:New> <https://launchpad.net/bugs/1632030>
[19:09]  * rick_h_ goes to get the boy home from school, biab
[20:23] <veebers> alexisb: Ina normal (test) run, should I see the log message: "ERROR cmd supercommand.go:458 creating API connection: ..."? (I'm seeing this error in the failing tests: ERROR cmd supercommand.go:458 creating API connection: EOF)
[20:24] <alexisb> not on a normal test run you are expecting to pass without error
[20:24] <veebers> I never see "creating API connection" in a passing test run (for grant-revoke)
[20:24] <alexisb> veebers, when did you start seeing the new failures?
[20:24] <veebers> alexisb: what is it a symptom of? One sec will check times
[20:25] <veebers> alexisb: looks to be about Oct 7 (Jenkins time, so UTC?) had 2 passes of 12 runs since then, the failures all include that error message
[20:28] <alexisb> menn0, ^^^
[20:28] <alexisb> I am wondering if this lines up with this merge: https://github.com/juju/juju/pull/6400
[20:32] <menn0> veebers: where are you see that message? from the juju client or in agent logs?
[20:33] <menn0> veebers: a certain amount of such messages are expected as when controllers come up and when workers restart after new controllers are added
[20:33] <veebers> menn0: um, I'm pretty sure the client? (I'm seeing it in the log output from juju command)  http://reports.vapour.ws/releases/4467/job/functional-grant-revoke/attempt/721 if you search for "connection: EOF"
[20:34]  * menn0 looks
[20:34] <veebers> menn0: this is after a controller comes up, users have been added etc.
[20:40] <veebers> menn0: Would this be related to the ping changes to? http://reports.vapour.ws/releases/4467/job/run-unit-tests-race/attempt/1949 (FAIL: monitor_internal_test.go:69: MonitorSuite.TestLaterPingFails)
[20:41] <menn0> veebers: that's a new test. i'll take a look.
[20:41] <menn0> veebers: the other issue appears to be macaroons related. here's the related bit in the controller's logs:
[20:41] <menn0> 2016-10-10 16:17:39 INFO juju.apiserver request_notifier.go:70 [1E] API connection from <IP_ADDRESS>:33086
[20:41] <menn0> 2016-10-10 16:17:39 INFO juju.apiserver admin.go:102 login failed with discharge-required error: verification failed: no macaroons
[20:41] <menn0> 2016-10-10 16:17:39 INFO bakery service.go:366 server attempting to discharge "eyJUaGlyZFBhcnR5UHVibGljS2V5IjoieVZlUEU3cUNJMm5nZWZuUXo2NlFPWVEwanlMVnVDanMwZGhMa0VUa3dRST0iLCJGaXJzdFBhcnR5UHVibGljS2V5IjoiNzF5NnNIOExNNUU1OHYxaGpGek9JdCtCZGRLRHR6SE5pN0ExQ1dBWHZqND0iLCJOb25jZSI6InpoMllTNnZ1bFFHNUlPdVRucXY4U0hVNW5MQWlSaE43IiwiSWQiOiJlSHM0Y3dERDRiOCs5TXBDUG1RakwvenVFdXZhV3dSbmhkY0tXWmtGQjA5MENaMXYxcnBYZ0RNczRsTTUrc2FSS2l2RGt0OFhlOG5BZTNxVG5
[20:41] <menn0> 4Y1YwMXhIUWZxTXRmWHk4TUdnNmRvSWZlZWpSZmJnby9pdjIxSmZiU2FkQjNwdXR6ZmJ2ekRvbGh1aDNPcGdCbTRQd2pSRGx4cGFEQnZaL0lwZGhoL0lQb3dMdXc9PSJ9"
[20:42] <menn0> 2016-10-10 16:17:39 INFO juju.apiserver request_notifier.go:80 [1E]  API connection terminated after 28.244335ms
[20:42] <menn0> 2016-10-10 16:17:39 INFO juju.apiserver request_notifier.go:70 [1F] API connection from <IP_ADDRESS>:33096
[20:42] <menn0> 2016-10-10 16:17:39 INFO juju.apiserver request_notifier.go:80 [1F] user-admin API connection terminated after 43.24719ms
[20:43] <menn0> veebers: i'll dig a bit more into that one to see if I can figure it out
[20:43] <veebers> menn0: I see a macroons change about 3 days ago, "remove macaroons on logout"
[20:44] <menn0> veebers: ok, could be related. those messages might also be normal. I'll do some digging.
[20:46] <veebers> menn0: Hmm, it's possible that the test misbehaves with those recent changes. Just looking now and if it's running on lxd (which this test is) it doesn't handle entering the password.
[20:47] <veebers> menn0: Did that behaviour change recently? (i.e. lxd not need password)
[20:47] <menn0> veebers: hmm, ok. that could be it.
[20:47] <menn0> veebers: I have no idea.
[20:47]  * veebers builds latest juju to test
[20:47] <veebers> menn0: let me test that and I'll get back to you
[20:47] <rick_h_> veebers: hmm, yea the branch nate did made sure to clear cookies when you logout
[20:47] <rick_h_> veebers: menn0 which meant that you had to actually login again after a logout for a change
[20:48] <veebers> rick_h_: that's probably it then, I should be able to confirm shortly
[20:49] <menn0> veebers: is there a ticket yet for the TestLaterPingFails intermittent failure?
[20:49] <veebers> menn0: not yet, I can make one right now
[20:50] <menn0> veebers: ok thanks. i've just reproed it here
[20:51] <menn0> grrr. even with injected clocks it's still too easy to introduce these intermittent timing issues ....
[20:52] <veebers> menn0: fyi https://bugs.launchpad.net/juju/+bug/1632105
[20:52] <mup> Bug #1632105: Test MonitorSuite.TestLaterPingFails fails  <ci> <regression> <unit-tests> <juju:Confirmed> <https://launchpad.net/bugs/1632105>
[20:53] <menn0> veebers: cheers
[21:14] <menn0> veebers: ok i've got the fix for TestLaterPingFails done... will propose shortly
[21:15] <veebers> menn0: sweetbix
[21:16] <menn0> veebers: thanks for pointing it out
[21:17] <veebers> menn0: heh, no worries, it popped up in the 'unknowns' that I'm wrangling
[21:18] <menn0> good to get on top of these intermittent failures quickly
[21:18] <veebers> agreed
[21:23] <menn0> thumper: easy review pls: https://github.com/juju/juju/pull/6420
[21:23]  * thumper looking
[21:24] <thumper> +1
[21:27] <veebers> menn0: ugh, sorry for the noise earlier, the fix for the grant/revoke/macroons stuff was in the CI test.
[21:27] <menn0> veebers: ok good to know
[21:40] <menn0> thumper: it's probably too late to expose the HighAvailabilty facade for controller logins isn't it?
[21:41] <thumper> menn0: yes and no
[21:41] <thumper> you could create a new one
[21:41] <thumper> but still need to support the old one
[21:41] <menn0> thumper: i'm working on the ticket to allow juju enable-ha to just work regardless of the current model
[21:42] <thumper> I guessed
[21:42] <menn0> thumper: the client side work is a piece of cake but you then end up with a controller login
[21:42] <menn0> thumper: the alternative is to do something tricky in the client so it logs into the controller model
[21:42] <thumper> Add the new facade
[21:42]  * thumper thinks
[21:43] <menn0> I don't think that would be hard to do and it's possibly lower risk
[21:43] <thumper> you need to have admin access in the controller model to do it
[21:44]  * menn0 checks what the current story with access is
[21:44] <thumper> or perhaps just write?
[21:45] <menn0> you currently need superuser access
[21:46] <menn0> that probably makes sense TBH
[21:50] <menn0> thumper: i don't *think* a new facade is required, the existing HighAvailability facade just needs to be exposed for controller logins and (controller) model logins (for compatibility)
[21:50] <thumper> ok
[21:50] <menn0> thumper: the client will have to try both approaches (in case a newer client is talking to an older server)
[21:50]  * thumper nods
[21:50] <menn0> thumper: with a preference for a controller login
[21:51] <menn0> ok
[21:51] <menn0> thumper: thanks rubber duck :)
[21:59] <veebers> thumper: you recently worked on some unit tests wrt to certificates/keys etc? Fyi I just filed this bug which may be of interest: https://bugs.launchpad.net/juju/+bug/1632127
[21:59] <mup> Bug #1632127: Unittest "MachineSuite.TestCertificateDNSUpdatedInvalidPrivateKey" fails on multiple archs <ci> <unit-tests> <juju:Confirmed> <https://launchpad.net/bugs/1632127>
[21:59] <thumper> veebers: as much as I love fixing bugs, I'm focusing on a different piece of work this week
[22:01] <veebers> thumper: ack, only wanted to bring it to your attention incase my assumption of you working in that area was correct :-)
[22:21] <menn0> veebers: TestCertificateDNSUpdatedInvalidPrivateKey was added by axw yesterday
[22:21] <menn0> it just failed for me on a merge attempt tooo
[22:23] <veebers> menn0: ack thanks, oh he's now on leave right?
[22:23] <menn0> veebers: ah yes... could be
[22:23] <menn0> veebers: assign it to me... I was working with him in that area yesterday
[22:23] <anastasiamac> menn0: veebers: Roger filed a bug for the failure.. i've marked Chris's as a duplicate...
[22:24] <menn0> anastasiamac: ok cool. bug number?
[22:24] <veebers> menn0: anastasiamac points out I filed a dupe bug
[22:24] <anastasiamac> menn0: veebers: https://bugs.launchpad.net/bugs/1631990
[22:24] <mup> Bug #1631990: cmd/jujud/agent: sporadic test failure in MachineSuite.TestCertificateDNSUpdatedInvalidPrivateKey <ci> <unit-tests> <juju:Triaged> <https://launchpad.net/bugs/1631990>
[22:24] <veebers> ah, beat me to it :-)
[22:25] <menn0> veebers: yep got that... I was after the original ticket
[22:25] <menn0> anastasiamac, veebers: thanks
[22:28] <jamespage> hey menn0 - I see you picked up bug 1541482
[22:28] <mup> Bug #1541482: unable to download local: charm due to hash mismatch in multi-model deployment <2.0-count> <juju-release-support> <uosci> <juju:Fix Released by menno.smits> <juju-core:Triaged by menno.smits> <https://launchpad.net/bugs/1541482>
[22:29] <menn0> jamespage: yep. I fixed the same/similar issue for 2.0 so I seemed like the right person to deal with it for 1.25.
[22:29] <jamespage> menn0, awesome
[22:29] <jamespage> menn0, its killing our final release testing atm
[22:29] <jamespage> menn0, any ideas what triggers the race that causes it?
[22:29] <jamespage> we don't see it all of the time
[22:30] <menn0> jamespage: IIRC it's to do with the apiserver's on disk charm cache
[22:30] <menn0> jamespage: the solution is to remove the cache. it's not necessary any more (already done for 2.0)
[22:31] <jamespage> menn0, yeah - I saw the fix you did for 2.0
[22:32] <menn0> jamespage: there might be some other related fixes in that area too. I need to review previous PRs.
[22:32] <jamespage> menn0, ok - I'll leave it with you
[22:32] <menn0> jamespage: actually, I just checked and I've already done that
[22:33] <menn0> jamespage: those changes just haven't been released yet
[22:33] <menn0> jamespage: we need a 1.25.7 release
[22:33] <jamespage> yp!
[22:34] <menn0> jamespage: if it would help I can supply binaries for you to use in the mean time (to validate that the problem is indeed fixed for you)
[22:36] <thumper> jamespage: how urgent is the need for 1.25.7 for your team?
[22:37] <jamespage> thumper, we release thursday; currently we're doing final sweep of amulet testing tidy including disabling old release combos and enabling new ones
[22:38] <jamespage> thumper, but that final sweep has been going for alot longer than normal as we can't get a full amulet run through consistently atm
[22:38]  * thumper nods
[22:38] <jamespage> thumper, we tried a workaround, but we're having to unpick that now
[22:39] <jamespage> thumper, as that did something quite different to what we expected
[22:39] <jamespage> thumper, so fairly urgent - I'll just see if we can splice in custom binaries to our Charm CI or not
[22:41] <thumper> jamespage: well, there is no way it is happening this week :(
[22:41] <thumper> so, sorry
[22:41] <jamespage> that was my guess
[23:18] <alexisb> menn0, ping
[23:22] <anastasiamac> jamespage: plz email alexisb and rick_h_re:urgency for 1.25.7 to come out \o/
[23:42] <mwhudson> alexisb: well i started on that bug and now i'm reading the kernel source :)
[23:45] <thumper> menn0: are you working on https://launchpad.net/bugs/1631990 now?
[23:45] <mup> Bug #1631990: cmd/jujud/agent: sporadic test failure in MachineSuite.TestCertificateDNSUpdatedInvalidPrivateKey <ci> <unit-tests> <juju:Triaged by menno.smits> <https://launchpad.net/bugs/1631990>
[23:47] <menn0> thumper: I haven't gotten to it yet
[23:48] <menn0> thumper: did you want to pick it up, or were you just bitten by it?


[01:12] <deluxux> damn gfx drivers keep turning off one of my screens
[01:12] <deluxux> or my intel gpu is maxxed out?
[02:41] <Compu> hey i need some help with theming, i set both a window border theme and a widget theme in the customize look and feel window but only some applications use it, firefox and gedit wont use it
[02:45] <Compu> even after a reboot


[03:17] <grantbow> no meeting tonight. Is anyone online from http://socallinuxexpo.org?
[03:17] <darthrobot`> [R: www.socallinuxexpo.org] Title: [SCALE 12x]
[03:21] <JoseeAntonioR> grantbow: I am
[03:23] <grantbow> cool! how's it going?
[03:34] <JoseeAntonioR> grantbow: all good, I'm checking a juju charm I'm writing right now
[03:34] <JoseeAntonioR> need to be here until 2am :P
[05:06]  * ianorlin is back from scale
[06:59] <rww> hihi, just got back home :)
[08:04]  * pleia2 home too
[08:06] <bkerensa> wow such home much sleep lol
[08:39] <nhaines> Whew, caught up with email and so on.  I'm going to unpack from SCALE tomorrow.  :P
[08:39] <rww> nhaines: you're not as popular as jose, he got 10 emails per keysigning person :D
[08:40] <nhaines> rww: I really should have signed up for that!  On the other hand I got to eat free pizza and drink free beer.
[08:40] <rww> on the other other hand, the pizza contained NSA nanobots that you could have combatted with the web of trust
[08:41] <rww> but yes, they allowed write-in fingerprints for people who didn't register with a fingerprint, so...
[08:41] <nhaines> I still need to sit down and generate another key. 16384 bytes should be enough for everybody, right?
[08:42] <rww> lol
[08:42] <rww> last one I went to someone had an 8k key
[08:43] <nhaines> ha
[08:46] <jose> urgh, that happened to me for having like 10 emails on my key
[08:46] <jose> a while ago I had like 50 unread emails - all with signed keys
[08:47] <rww> jose: did you try that keychain from aaron yet? doesn't seem to be working for me
[08:47] <jose> rww: you mean the keyring?
[08:47] <rww> yes
[08:47] <jose> if you give me a minute I can go ahead and try
[08:48] <jose> rww: are you using seahorse or something like caff?
[08:49] <rww> pius says "Failed to find keys on this keyring", and gpg --no-default-keyring --keyring keyring.gpg --list-keys doesn't list anything
[08:50]  * jose checks
[08:52] <jose> rww: it does show an output for me, can you try putting ~/path/to/file/keyring.gpg instead of just keyring.gpg?
[08:52] <jose> that made the trick for me
[08:52] <rww> ... ohhhhhh, that triggered memories of me doing this in the past
[08:52] <rww> thanks :)
[08:52] <jose> sure :)
[08:53] <jose> nhaines: if you have a key you want me to sign I can just do that right ahead
[08:53] <jose> just scan two photo id proofs, send a paypal payment to<EMAIL_ADDRESS>for $20 for the concept of handling, and you're all set
[09:11] <jose> rww: thank you!
[09:20] <rww> jose: btw, you have two uids on your gpg key that have invalid (bounced back) emails on them. might want to revoke those uids at some point
[09:20] <jose> rww: two? I know<EMAIL_ADDRESS>is not working anymore, but what's the other one?
[09:21] <jose> ooooooooooh
[09:21] <jose> I just found one misspelled
[09:21] <rww> the ubuntu-pe.org one?
[09:21] <jose> ye[
[09:21] <jose> yep*
[09:21]  * rww nods
[09:21] <jose> should be joseeantonior
[09:21] <jose> I don't know if I should revoke it or if I should just create an alias to my current email
[09:22] <jose> I have admin access to the gapps panel for that domain :P
[09:32] <rww> hrm, odd. pius doesn't like one of atoponce's uids
[09:32] <jose> rww: which one?
[09:32] <rww> the xmission one
[09:33] <jose> lemme check
[09:33]  * rww tries it manually to see why
[09:36] <jose> hmm, looks like it's worked for some people
[09:39] <rww> hrm, it's off-by-one on the numbering for that uid
[09:40] <rww> aanndd I have an old version of pius. guess I'll try a newer one
[09:40] <jose> :P
[09:40] <rww> to be fair, this is Debian's fault, not mine :P
[09:40] <rww> lol, the maintainer is someone I've keysigned with
[09:41] <jose> :P
[09:41] <jose> aren't you on ubuntu?
[09:43] <rww> yes, but Ubuntu pulls the package from Debian.
[09:43] <rww> yeah, works fine in current version
[09:43] <rww> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=681939 probably just needs a poke
[09:43] <darthrobot`> Title: [#681939 - new upstream version available - Debian Bug report logs]
[09:44] <jose> :P
[09:45] <rww> ... and the email address that it was failing for won't receive email from me because xmission has decided in their infinite wisdom to blacklist me
[09:46] <rww> and their blacklist page very helpfully tells me that I'm blacklisted and then gives an error when I try to fix that
[09:50] <jose> well, maybe you're in some global blacklist
[09:50] <jose> ?
[09:50] <jose> that happened to trekweb.org a while ago
[09:51] <rww> I'm not.
[09:51] <jose> hmm, ok then
[09:51] <jose> I need to wait for another ten minutes in order to catch the airport shuttle
[09:51] <rww> probably atoponce forwards all his mail there and it noticed I sent a lot of similar stuff recently and temporarily nuked it
[09:52] <rww> anyways, I poked him on IRC and will take another look tomorrow morning
[09:52] <jose> cool
[09:54] <jose> rww: again, it was a pleasure to finally meet you :)
[09:54] <nhaines> jose: have a safe flight!
[09:54] <jose> I have to run and grab my baggage from storage to take that shuttle
[09:54] <jose> nhaines: thank you, and I'm looking forward to working with you in the near future - expect a couple emails soon! :)
[09:55]  * jose runs while falling asleep
[18:10] <pleia2> Corey: OH, hah, I just got the Ricardo reference :) (I don't check work email much when I travel, and that's what is hooked up to my account)
[18:23]  * ianorlin is sad he didn't really get to meet corey
[18:25] <pleia2> ianorlin: did you see his upscale lightning talk? was pretty funny :)
[18:26] <ianorlin> no I was in the crunchband BOF
[18:26] <ianorlin> I regret not now
[18:29] <pleia2> I think they were recorded
[18:33] <ianorlin> although I watched a video of jono bacon of last years scale and there was something wierd with the video in a disappering
[18:38] <Corey> ianorlin: Sorry, did we meet this weekend?
[18:38] <Corey> I was sorta pulled in a LOT of directions at once. :-)
[18:39] <pleia2> Corey: I'm totally stealing your dns joke
[18:39] <Corey> pleia2: Hahaha...
[18:40] <Corey> Yeah, it was too easy.
[18:40] <pleia2> indeed
[18:40] <Corey> pleia2: This week is a bit nutty, but ideally whisky again?
[18:41] <pleia2> Corey: that would be fun, but ask me again when you're available, right now I'm so snowed under that I don't think I'll be able to leave my desk for about 7 weeks :)
[18:41] <pleia2> hooray for Monday after conference
[18:42] <Corey> Yeah, I'm suffering from ConFlu.
[18:42] <pleia2> so is mjoseph :(
[18:42] <blitz> do all of you work in san francisco?
[18:43] <pleia2> Corey comes here a lot for work, and I do
[18:43] <pleia2> blitz: you come out for whiskey too
[18:43] <blitz> only if I can have beer instead
[18:43] <blitz> I shame my scottish roots
[18:43] <pleia2> hehe
[18:44] <Corey> blitz: Beer is fine.
[18:44] <pleia2> Corey: I think it was the beer and sake before all the whiskey that got me in trouble last time :)
[18:47] <pleia2> Corey: if I buy all the drinks, can I get a GC for openstack?
[18:48] <ianorlin> no we did not
[18:48]  * pleia2 checks corporate policy for expensing this, and is disappointed
[18:49] <blitz> im going to be at the aws conference if anyone's going to that
[18:50]  * pleia2 hides under her openstack rock and avoids snarky aws comments
[20:02] <rww> jose: You too. Hope your flight went well :)
[20:04] <rww> I don't have conference flu (yet)!
[20:05]  * ianorlin is feeling good as well
[20:16] <nhaines> My throat was a little raw this morning but it appears to have cleared up.
[22:41] <nhaines> jono: the details on the new scopes infrastructure looks good... I'm just sorry I have to wait until October to see it on the desktop too.  :)
[22:41] <jono> nhaines, thanks!
[22:43] <raevol> i wrote a report back to my team on how the openmw talk went, dunno if anyone here would be interested, but just in case: https://forum.openmw.org/viewtopic.php?f=4&t=2053
[22:43] <darthrobot`> Title: [openmw.org • SCaLE 12x Talk Debrief]
[22:49] <nhaines> raevol: I didn't know there was an openmw talk, and I didn't know you were involved.  But I'm super interested to see how it went!
[22:49] <raevol> :D
[22:49] <pleia2> raevol: nice :)
[22:51] <nhaines> raevol: just finished the article.  Looks like it went really perfectly!
[22:52] <pleia2> I have no idea what people asked at mine, it's all a blur, kudos
[22:59] <raevol> thanks thanks :D i had a really good time. i feel like i want to do more speaking, haha, it was really fun
[23:00] <pleia2> you should
[23:02] <nhaines> raevol: definitely do it.  :)
[23:02] <nhaines> pleia2: I saved all my griefer questions for jono.  :)
[23:03] <pleia2> haha
[23:04] <nhaines> Actually (except for the text editor one) my questions for him were sincere and I was impressed by the answers.


[01:22] <jadestar> how do i change key layout
[01:22] <jadestar> in openbox :T
[01:22] <jadestar> sorry
[01:22] <jadestar> system key layout XD
[01:50] <phillw> jadestar: what version of lubuntu are you running?
[01:51] <jadestar> lightweight ubuntu extended life edition or LXLE
[01:52] <jadestar> based on lightweight ubuntu 12.06
[01:52] <phillw> jadestar: in that case http://ubuntuforums.org/showthread.php?t=1455877 should sort it out
[01:52] <jadestar> nm
[01:52] <jadestar> misnumbered
[01:52] <jadestar> should be four hehe
[01:52] <jadestar> y thank u phillw :>
[01:52] <phillw> jadestar: it was close enough :)
[01:53] <jadestar> i use openbox
[01:53] <jadestar> L X L E has its own channel though
[01:55] <phillw> afaik, the keyboard settings are done under the lxde settings, not by open box. open box only does the DE, not play with keyboards.
[01:55] <jadestar> ah dhyanavadaha
[01:55] <jadestar> or
[01:55] <jadestar> uh
[01:56] <jadestar> thank you
[02:59] <katkisson> hi, I'm trying to watch a Hulu movie online and they say I need the newest version of Flashplayer. i thought that was downloaded when I  installed Lubuntu a few days ago.
[03:02] <phillw> !flash
[03:02] <phillw> katkisson:  ^^
[03:07] <katkisson> Hi, I was just in Ubuntu asking about Flash Player. I tried to go to Hulu online to watch a movie, and they said to install the latest version. Someone told me a command to put in terminal, but it didn't find that command.
[03:09] <katkisson> o.k. I'll try that site right now.
[03:12] <katkisson> Now, I see Adobe Flash Plug-in in Lubuntu software. Will that work/
[03:13] <katkisson> It says it's for Mozilla Firefox. That
[03:14] <katkisson> fine. I downloaded that.
[03:16] <katkisson> I don't have Java. Should i get that?
[03:31] <katkisson> Just so you know. I found Flash Player Plug in  in the Lubuntu software and downloaded it. It's works. I can watch Hulu movies now. Yeah!
[03:31] <holstein> !flash
[03:31] <holstein> should work fine..
[03:32] <katkisson> thanks!
[06:01] <micahg> I'm thinking to updating libgoffice and gnumeric for saucy, any objections?
[08:53] <manik> hey
[08:53] <manik> Can anybody tell me a file manager which gives similar search option like Windows Explorer?
[09:54] <New2> hi friends
[09:54] <New2> google gives Firefox $400 million per year to data mine your habits, i found this to help get rid of google from firefox http://www.leavegooglebehind.com/how-tos/how-to-eradicate-google-from-firefox/
[09:56] <New2> "Firefox has a feature called “safe browsing” which reports information about your browsing to Google."
[09:56] <New2> oops wrong channel sorry lol
[15:45] <jeepsy> i cannot login in the main account anymore. at first i thought problem was password even though i was quite sure i was doing it right. then i noticed that when i purposely put a wrong password i do get a wrong password message. instead when i put the good password, i dont get that message. i get a black screen for a sec and then back to login screen. what's going on?
[15:46] <bioterror> ctrl+alt+f1
[15:46] <bioterror> login from the console
[15:46] <bioterror> and you will be a little more smarter
[15:46] <bioterror> if that works, then start-lubuntu or what was it
[15:46] <bioterror> startlubuntu
[15:47] <jeepsy> what do i type to login from the console?
[15:48] <bioterror> Wed18:46 <bioterror> ctrl+alt+f1
[15:54] <bioterror> so?
[15:55] <jeepsy> ok login works. but when i do startlubuntu it doesnt. i get something like "gtk warning cannot open display" and above iread many permission denied errors
[15:55] <bioterror> and what's the last thing you have done before it went broken?
[15:56] <jeepsy> nothing. updates i guess
[15:58] <jeepsy> it's totally gotten me by surprise. how do you say "out of the blue"
[15:58] <jeepsy> no reason i can think of
[16:00] <jeepsy> any idea of what i should do?
[16:02] <bioterror> be more specific
[16:02] <jeepsy> i just used the computer regularly for regular stuff, libreoffice, mail, you know. all of a sudden i get this error
[16:03] <jeepsy> what do you want to know?
[16:05] <bioterror> the permissions, ofcourse
[16:05] <bioterror> where you do not have permissions
[16:05] <jeepsy> only thing i can think of that messed it up are the updates. but i guess that doesnt make much sense. updates solve bugs, they dont create them. i guess
[16:05] <bioterror> :D
[16:06] <jeepsy> //config if remember right
[16:06] <bioterror> how naive
[16:07] <jeepsy> i am a newbie :)
[16:07] <jeepsy> how can i show you that output?
[16:07] <jeepsy> i am now using the guest session which works fine
[16:08] <bioterror> you're using the same computer with same installation
[16:08] <bioterror> with guest account?
[16:08] <jeepsy> yes
[16:08] <Unit193> So, move /home/username-here/.config/ to somewhere else and then try it.
[16:08] <bioterror> yep
[16:08] <Unit193> Remove .cache/ too while you're at it, saved sessions can make a day fun.
[16:09] <jeepsy> can you please tell me step by step (help for dummies)
[16:09] <bioterror> login with that ctrl+alt+f1
[16:09] <bioterror> then
[16:09] <bioterror> mv .config .config.bak
[16:09] <bioterror> rm -rf .cache
[16:10] <bioterror> you do not need to logout
[16:10] <bioterror> now
[16:10] <bioterror> you can just ctrl + alt + f1
[16:12] <jeepsy> ok will try that. but i have a little unxpected problem here (not with the computer) that i must take care of now. will do it later. thank you both for the help
[16:37] <thor_> Hi Please a link to the meeting log from yesterday


[02:23] <squish102> wow superfly, i'm glad i dont have pets
[02:23] <squish102> i heard it was expensive


[02:13] <mason> So, I see Nehalem and Nehalem-IBRS cooked into qemu-2.11+dfsg/target/i386/cpu.c.
[02:16] <mason> I wonder if my host is too old to have CPUID_7_0_EDX_SPEC_CTRL or something. I'm not 100% sure how that works as yet.
[02:20] <mason> I don't know. I migrate the thing over, and  Nehalem-IBRS no longer shows up in the list of possible models, but it does show up as the active CPU model. Weird.
[02:20] <mason> Oh well.
[09:25] <cpaelzer> mason: those are libvirt specific types
[09:26] <cpaelzer> mason: for qemu that is some-base-cpu+feature-bits
[09:26] <cpaelzer> libvirt does host probing for capabilities, so maybe due to that it differs
[09:26] <cpaelzer> well your question was so many hours ago, did you find anything already?
[13:53] <mason> Ooh, here's an opportunity to use the memo-bot!
[22:47] <kinghat> why am i now getting "kinghat is not in the sudoers file.  This incident will be reported." on my server?
[22:47] <kinghat> was there a recent update that changed something? never seen this before except for on fedora.
[22:56] <ikonia> kinghat: you have changed either the sudoers file, or you're users group membership
[23:01] <kinghat> hmm
[23:03] <kinghat> i think i added the user to the group `web-content` for apache, would that have done it?
[23:03] <kinghat> i thought the user could be apart of multiple groups?
[23:04] <ikonia> kinghat: I suspect you've added the user to the group and at the same time removed it from the admin / wheel group
[23:04] <ikonia> yes, a user can be a member of many groups
[23:04] <ikonia> type "id" against your user
[23:04] <kinghat> uid=1000(kinghat) gid=1000(kinghat) groups=1000(kinghat),1001(web-content)
[23:04] <kinghat> ya i was just going looking for the commands to get a table of users and groups on the system.
[23:09] <kinghat> ikonia: so you are saying i need to get root, then add my user to `sudo` user group?
[23:09] <ikonia> what groups is your user in ?
[23:09] <kinghat> because i cant add my account via sudo atm because its not part of sudo.
[23:10] <kinghat> i posted it above i thought.
[23:10] <ikonia> oh yes
[23:10] <ikonia> so boot into single user mode and add your user to the sudo group
[23:10] <kinghat> single user mode = root?
[23:11] <ikonia> what ?
[23:11] <kinghat> what is single user mode?
[23:11] <ikonia> it boots the user into just that - single user mode, rather than the multi-user mode and permissions system you currently have
[23:12] <ikonia> or you can just become root if you've set a root password and know it
[23:13] <kinghat> i feel like ive logged into root using the same password as the initial account before w/o having set an actual root password.
[23:14] <ikonia> then su -
[23:14] <kinghat> ya i just did that and it says su: Authentication failure.
[23:14] <ikonia> then you don't know the password
[23:15] <kinghat> if anything i made it the same as the initial account. actually i figured thats what the installer did by default.
[23:17] <ikonia> nope
[23:17] <ikonia> it does what you tell it to do
[23:18] <kinghat> so i have to give root a password before i can use root?
[23:18] <ikonia> yes
[23:18] <ikonia> (the correct model is to not set a root password and have a sane sudo setup)
[23:19] <kinghat> sure but how do i get my only user back to sudo then?
[23:20] <ikonia> boot into single user mode
[23:20] <kinghat> can that be done via ssh?
[23:20] <ikonia> possibly, but it normally requires console access as single user mode is not on the network
[23:22] <kinghat> why would adding this user to another group take it out of another group?
[23:22] <ikonia> depends how you added it
[23:22] <kinghat> sudo adduser username groupname
[23:22] <ikonia> kinghat: that's creating a new user
[23:22] <ikonia> that's not adding a user to a group
[23:23] <kinghat> sorry it was this: `usermod -G web-content alice`
[23:23] <kinghat> but `kinghat` over alice.
[23:24] <kinghat> and i also added apache to that group.
[23:24] <ikonia> so that's just told the system to put the user kinghat in the group web-content only
[23:24] <ikonia> that's not adding a user to a group, that's setting a user to that single group
[23:24] <kinghat> well that explains it
[23:24] <ikonia> I suggest you read the man page of commands if you don't know how to use them
[23:25] <kinghat> https://wiki.apache.org/httpd/FileSystemPermissions
[23:25] <ikonia> ?
[23:25] <ikonia> what has that got to do with anything ?
[23:25] <kinghat> i feel like there should be a warning or something. are you sure you want to remove this user from all groups? especially the sudo group.
[23:25] <ikonia> no
[23:26] <ikonia> you should read the man page of commands you don't know about, more so if executing them with root access
[23:26] <ikonia> the man page is really clear on the syntax
[23:26] <kinghat> well now i have to figure out how to get single user w/o console access.
[23:27] <kinghat> what about giving root a password then doing what i need and then removing the password from root?
[23:27] <ikonia> I suspect that will be exceptionally hard
[23:27] <ikonia> how do you plan to set a root password without having root access ?
[23:27] <kinghat> you mean w/o being apart of the sudo group? ya i was just realizing that.
[23:28] <kinghat> fak
[23:28] <ikonia> please control the languge
[23:29] <kinghat> so basically i cant really do anything w/o having physical access?
[23:29] <ikonia> you'll find it exceptionally hard
[23:29] <ikonia> and high risk
[23:30] <kinghat> apparently im high risk.
[23:30] <ikonia> what ?
[23:31] <kinghat> im the risk.
[23:31] <ikonia> clearly
[23:31] <OerHeks> no direct access makes it just a little complicated
[23:35] <kinghat> you have to edit grub?
[23:36] <ikonia> that is one of the high risk ways, however if you did it right and it boots into single user mode, how will you interact with it ?
[23:38] <kinghat> no im asking thats how you get into single user mode?
[23:39] <ikonia> right, there are a few options, but what's the plan if you did get it into single userm ode
[23:39] <kinghat> you said add my user to `sudo`?
[23:39] <ikonia> how will you do that remote
[23:41] <kinghat> i dont have physical access atm but i will have to get physical access apparently so.
[23:42] <ikonia> so I'd deal with that when you get access as the access you get / how you get it will guide you the best way to get access to the root user
[23:44] <kinghat> i mean i have to go there to get physical keyboard/monitor console access. its going to happen once i figure out how its done before i go.
[23:45] <kinghat> aka im going as soon as i figure out how to get single user mode.
[23:52] <kinghat> so adding a user to another group and not ONLY this group is: `sudo usermod -a -G sudo username`?