[04:52] <fishcooker> should i restart the box after configure network interfaces? http://vpaste.net/HQirT AFAIK im used to do /etc/init.d/networking restart or ifconfig eth0 down && ifconfig eth0 up, but it doesn't work
[08:55] <lordievader> Good morning.
[09:32] <rbasak> cpaelzer: did you intend to drop the openldap merge from the blueprint?
[09:38] <cpaelzer> rbasak: yeah it was a dup, check the second to last line
[09:38] <cpaelzer> rbasak: I was updating mine and picked a few of the more simpler merges to fill into my qemu testing idle time slots
[09:38] <cpaelzer> rbasak: and so I found a few things that were duplicates or not updated yet
[09:39] <cpaelzer> rbasak: I hope there is no openldap != openldap magive that I'd have missed - did I ?
[09:56] <rbasak> cpaelzer: ah OK. Just checking!
[10:25] <saju_m> In what order
[10:25] <saju_m> In what order I should install these packages http://dpaste.com/1Q388JC ?
[13:15] <zul> coreycb: i uploaded a newer tempest last night btw
[13:16] <coreycb> zul, ok thanks
[13:42] <zioproto> coreycb: can you confirm that Openstack Kolla in not yet package for Openstack Newton in Ubuntu ?
[13:44] <coreycb> zioproto, confirmed.  although we were talking about packaging it.
[13:52] <zioproto> ok, I am testing it at the moment, right now working with pip then
[13:54] <jamespage> zioproto, hey
[13:54] <zioproto> hey
[13:54] <jamespage> zioproto, are you looking at kolla itself or one of the kolla-ansible/k8s deployment projects?
[13:55] <zioproto> we have this problem here, that we dont feel like rewriting our puppet stuff from puppet3 to puppet4
[13:55] <zioproto> at the moment looking at kolla in stable/newton
[13:55] <zioproto> that is still just kolla without the additional kolla-ansible repo
[13:55] <zioproto> that starts in ocata
[13:55] <zioproto> kolla-ansible repo only has a master branch
[13:56] <jamespage> zioproto, ack
[13:56] <zioproto> still did not look at Kubernetes
[13:56] <jamespage> zioproto, ok so can I suggest a slight different approach to packaging kolla then ?
[13:57] <zioproto> sure
[13:57] <zioproto> I am all ears !
[13:57] <jamespage> zioproto, have you heard of snaps?
[13:57] <zioproto> is k8s different from https://github.com/openstack/kolla-kubernetes ?
[13:57] <jamespage> kubernetes == k8s
[13:57] <jamespage> for folk like me that can't spell or be bothered to type a long name :-)
[13:57] <zioproto> ok, so this is the same stuff ? https://launchpad.net/kolla-k8s
[13:58] <jamespage> yah
[13:59] <zioproto> so what different approach you would suggest ?
[13:59] <zioproto> ah snaps
[13:59] <zioproto> yes
[13:59] <jamespage> zioproto, yeah
[13:59] <zioproto> yes
[13:59] <jamespage> zioproto, https://github.com/openstack-snaps/snap-rally
[13:59] <zioproto> packaging distribution agnostic, right ?
[13:59] <jamespage> yep
[13:59] <zioproto> I hear of it
[13:59] <zioproto> but I dont have experience
[13:59] <jamespage> zioproto, we've been looking at it alot in the last few months
[13:59] <jamespage> zioproto, have a mostly functional set of openstack snaps
[14:00] <zioproto> actually is how we deploy RC packages on Cumulus Linux
[14:00] <jamespage> I just raised the review to get rally and tempest snaps up under /openstack
[14:00] <joelio> jamespage: interesting, I'm working on rally today. Code available?
[14:00] <zioproto> do you have a stable/newton openstack snap for Kolla that I can test ?
[14:00] <jamespage> zioproto, not yet - I was going to suggest that you have a run at it
[14:01] <zioproto> jamespage: are you coming to Milano to the ops meetup ?
[14:01] <jamespage> they are pretty quick to write
[14:01] <jamespage> zioproto, might be
[14:01] <jamespage> zioproto, plans tbc
[14:01] <zioproto> okay, so I will first give it a try with pip
[14:01] <jamespage> joelio, https://github.com/openstack-snaps/snap-rally
[14:01] <joelio> thanks
[14:01] <jamespage> is installable from the snap store - take a read of the README
[14:01] <zioproto> Kolla is already new, and I cant introduce in the team 3 new technologies at the same time :)
[14:02] <jamespage> zioproto, just think of it as a static wrap of pip
[14:02] <jamespage> installable without infecting the rest of your install with pip-ness
[14:03] <jamespage> joelio, fwiw I think you need the snapd from xenial-proposed for the rally snap - its a classic mode version
[14:03] <jamespage> which is pretty new feature
[14:04] <jamespage> joelio, 2.21 is the version
[14:04] <joelio> jamespage: yea, I'll take a look - I'd be keen to look at this for the cli tooling too, currently we provide a Vagrantfile for customers and a web interface, a snap would be cool too
[14:04] <jamespage> joelio, oh I already have one in flight
[14:04] <joelio> ace :)
[14:04] <jamespage> joelio, openstackclients
[14:04] <jamespage> joelio, idea is that it provides all openstack project cli tools
[14:04] <joelio> brill, sounds ideal
[14:05] <jamespage> the snap store is due to support series soon as well, so we can have a openstackclients aligned with each stable release of openstack
[14:05] <joelio> awesome
[14:06] <jamespage> joelio, we're bootstrapping snap bits atm - if you wanted to join #openstack-snaps that's where most irc discussion is happening
[14:07] <joelio> jamespage: ack
[14:07] <jamespage> zioproto, ^^ if you're interested as well :-)
[14:13] <zioproto> ok ! I will try to join also there
[14:13] <zioproto> so when I run commands like
[14:13] <zioproto> sudo snapper create -d "Installing DHCPd pre-release fix" --command "dpkg -i ./isc-dhcp-*deb"
[14:13] <zioproto> I am already use snap
[14:13] <zioproto> is the same tool right ?
[14:13] <zioproto> or snapper is yet another thing ?
[14:31] <joelio> zul: different, thats for snapshotting your local stuff.. this is more akin to an image registry/catalogue
[14:31] <joelio> zioproto: ^^ (sorry zul)
[14:31] <joelio> I just tested rally out, took less than a minute including setting up xenial-proposed, lgtm
[14:35] <smoser> dannf`, please feel free to verify bug 1640519
[14:52] <zioproto> joelio: so we are talking about this ? http://snapcraft.io/
[14:52] <joelio> yep
[15:05] <dannf`> smoser: yep - i'll ask sfeole to try it in his setup
[15:23] <cpaelzer> rbasak: could you run the importer on tgt ?
[15:23] <cpaelzer> rbasak: currently old/new debian is still outdated so it needs an import anyway
[15:23] <cpaelzer> rbasak: but I'm eager to see what it does since this already has a zesty merge
[15:23] <cpaelzer> yet I'm considering a re-merge to pick up latest (and drop more delta IIRC)
[15:26] <jamespage> zioproto, have you considered switching to charms from your current puppet approach for deployment of openstack?
[15:27] <zioproto> mmm… no, not at all :)
[15:31] <rbasak> cpaelzer: sure
[15:32] <cpaelzer> thanks rbasak
[15:36] <rbasak> cpaelzer: done
[16:14] <coreycb> EmilienM, is ocata-proposed working ok for you know that webob is back to 1.6.2?
[16:15] <EmilienM> coreycb: I haven't tested
[16:15] <EmilienM> mwhahaha: ^
[16:15] <mwhahaha> no
[16:15] <mwhahaha> we've got other issues
[16:15]  * mwhahaha hasn't had time to dig
[16:15] <mwhahaha> but nova's broken
[16:16] <coreycb> mwhahaha, in what way?
[16:17] <mwhahaha> coreycb: http://logs.openstack.org/66/422766/4/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial-nv/790428e/console.html#_2017-01-24_15_07_22_546980
[16:17] <mwhahaha> https://review.openstack.org/#/c/422766/ if you want to follow along
[16:18] <coreycb> mwhahaha, is there any chance that run has the old webob?
[16:18] <mwhahaha> nope
[16:18] <coreycb> mwhahaha, hmm
[16:18] <mwhahaha> http://logs.openstack.org/66/422766/4/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial-nv/790428e/logs/dpkg-l.txt.gz
[16:18] <mwhahaha> 1:1.6.2-2~cloud0
[16:19] <coreycb> mwhahaha, any details in the scheduler log as to why no valid host was found?
[16:19] <mwhahaha> http://logs.openstack.org/66/422766/4/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial-nv/790428e/logs/
[16:20] <mwhahaha> like i said, i haven't looked deeply yet. there was a vif error that is saw when i took a few seconds
[16:21] <mwhahaha> http://logs.openstack.org/66/422766/4/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial-nv/790428e/logs/nova/nova-compute.txt.gz#_2017-01-24_15_01_25_380
[16:27] <jamespage> mwhahaha, coreycb: hold a mo
[16:27] <jamespage> I think that might be due to a os-vif incompat we had
[16:27] <jamespage> Rebuilding amd64 build of nova 2:15.0.0~b2-0ubuntu2~cloud0 in ubuntu xenial RELEASE
[16:27] <jamespage> just triggered that
[16:27] <coreycb> jamespage, hmm
[16:27] <jamespage> sorry that fell off my plate yesterday
[16:28]  * mwhahaha buys jamespage a bigger plate :D
[16:28] <coreycb> :)
[16:30] <jamespage> coreycb, mwhahaha: hmm
[16:30] <jamespage> coreycb, mwhahaha: no I did do it yesterday
[16:30] <jamespage> 2:15.0.0~b2-0ubuntu3~cloud0
[16:30] <jamespage> is the right version
[16:31] <jamespage> we might have a general os-vif compat issue in that case
[16:34] <mwhahaha> http://logs.openstack.org/66/422766/4/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial-nv/790428e/logs/nova/nova-conductor.txt.gz#_2017-01-24_15_00_57_582
[16:46] <jamespage> mwhahaha: [Errno 2] No such file or directory
[16:46] <jamespage> looks like something fails on the compute node with regards to port creation
[16:47] <coreycb> jamespage, yeah would be nice to know what that file or director is
[16:47] <jamespage> 2017-01-24 15:01:52.220 4178 DEBUG oslo.privsep.daemon [-] u'brctl addbr qbr9926d47b-cb' failed. Not Retrying. out_of_band /usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py:194
[16:47] <jamespage> brctl by the looks of things
[16:48] <mwhahaha> http://logs.openstack.org/66/422766/4/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial-nv/790428e/logs/syslog.txt.gz
[16:48] <mwhahaha> there's the syslog if you want to go looking
[16:49] <mwhahaha> looks like there's a bunch of aodh errors too we'll need to cleanup at some point
[16:50] <jamespage> mwhahaha, coreycb: indeed bridge-utils is not installed
[16:53] <jamespage> mwhahaha, coreycb: ok looks like a dep change in libvirt is causing this
[16:53] <jamespage> xenial libvirt has a hard depends libvirt-bin -> bridge-utils
[16:53] <jamespage> in ocata uca
[16:54] <jamespage> libvirt-daemon-system -> recommends bridge-utils only
[16:54] <mwhahaha> pesky libvirt
[16:55] <mwhahaha> if you're fixing deps, python-gabbi is also required by tempest. we fixed it in our testing since we don't use the packages but it wasn't getting pulled in
[16:56] <coreycb> mwhahaha, ah ok we'll take a look at that
[16:56] <jamespage> coreycb, we don't see this as juju install bridge-utils automatically on all units for container bridging and addresssing
[16:56] <jamespage> but it does need a tweak
[16:57] <jamespage> I'd suggest actually having the dep on neutron-openvswitch-agent or nova-compute - not quite sure which is best tbh
[16:57] <jamespage> nova uses brctl directly I think
[16:58] <jamespage> coreycb, prob nova-compute methinks
[16:59] <mwhahaha> for what it's worth, rdo has nova-compute -> bridge-utils
[16:59] <mwhahaha> https://github.com/rdo-packages/nova-distgit/blob/rpm-master/openstack-nova.spec#L144
[16:59] <coreycb> jamespage, mwhahaha: yeah seems to make sense to add as a dep for nova-compute
[16:59] <coreycb> we only have it for nova-network atm
[17:00] <coreycb> alright let me make those 2 updates
[17:10] <coreycb> mwhahaha, it doesn't look like tempest uses gabbi. could it be a project-specific plugin?
[17:11] <mwhahaha> it's possible
[17:12] <mwhahaha> coreycb: ceilometer test
[17:13] <mwhahaha> http://logs.openstack.org/48/422248/8/check/gate-puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial-nv/1c85705/console.html#_2017-01-20_22_23_32_199166
[17:30] <coreycb> mwhahaha, ok i'll add that dependency for ceilometer.  these will build and i'll let you know when they're backported and promoted to ocata-proposed.
[17:30] <mwhahaha> coreycb: thanks
[17:40] <greyolla> Is there a way I can do my own live kernel patching without using canonical's livepatch service? ie looking to self host/maintain.
[17:56] <jgrimm> caribou, do you still have merges in progress for kdump-tools, clamav, and nut packages?  thought i'd make sure they are still on your radar
[17:57] <jgrimm> oh, kdump-tools was for secureboot, not a merge
[17:57] <caribou> jgrimm: kdump-tools/makedumpfile is synced with Debian; clamav is awaiting on the MIR of tomsfastmath to become a sync
[17:58] <caribou> jgrimm: nut still needs to be completed afaik
[17:58] <jgrimm> cool enough, i was just going through the blueprint and checking in on INPROGRESS items
[17:59] <caribou> jgrimm: I'd like to get the kexec-tools done too depending on my b/w
[17:59] <jgrimm> ack
[18:13] <JemalMoha> Hello All, nice to be part of the community!
[18:19] <ctjctj> Hello.  I just upgraded to 16.10 and iscsitarget no longer exists.  What is the replacement package?
[18:33] <Noname01x2> i need assistance
[18:34] <ctjctj> Noname01x2, ask your question. don't ask for attention.
[18:35] <Noname01x2> ok
[18:36] <Noname01x2> so on my ubuntu server i have apparmor, which i need. and its not working. getting an error in the log.
[18:37] <Noname01x2> im open to any solution to get rid of the error, although i would prefer not to remove mysql-server in the process.
[18:37] <sarnold> please pastebin your DENIED entries
[18:37] <Noname01x2> how do i do that? im no pro
[18:38] <Noname01x2> i took a screenshot
[18:39] <Noname01x2> saved it in paint
[18:39] <sarnold> Noname01x2: dmesg | grep DENIED should do it
[18:40] <sarnold> you could use the pastebinit program to automatically copy-and-paste that to a pastebin site
[18:41] <sarnold> ctjctj: I don't know about _the_ replacement for iscsitarget, but there's a few choices.. both libsiscsi and tgt appear to be in main in 16.10: http://packages.ubuntu.com/source/yakkety/tgt  http://packages.ubuntu.com/source/yakkety/libiscsi
[18:41] <Noname01x2> hmm let me try that
[18:41] <ctjctj> tgt and libiscsi and I'm checking istgt now.  All my google foo is pulling up iscsitarget references.  THanks sarnold.
[18:44] <Noname01x2> im in emergincy mode
[18:45] <ctjctj> Noname01x2, *grins* I understand.  I've got 10+ devs that are grumping because we lost all of our iscsi targets.
[18:46] <sarnold> owww
[18:47] <ctjctj> I'll blame you for the solution.
[18:48] <sarnold> when devs are grumbling is probably not the ideal time to be doing research, but last time I looked into iscsiland, this table seemed useful http://scst.sourceforge.net/comparison.html
[18:49] <noname01x3> sending you a file, please accept sarnold
[18:49] <sarnold> eh?
[18:49] <noname01x3> its a screenshot
[18:49] <sarnold> why not just pastebin the dmesg | grep DENIED?
[18:49] <noname01x3> cant
[18:49] <noname01x3> let me try again
[18:50] <noname01x3> im in the emergency mode terminal thing
[18:51] <noname01x3> i coppied the msg
[18:51] <noname01x3> now what
[18:52] <ctjctj> noname01x3, if you don't have it installed, install pastebinit   and use it to create pastebin documents.  You'll get back a URL which you post here.
[18:52] <ctjctj> !pastebin
[18:53] <noname01x3> im rebooting
[18:53] <Noname01x2> ohh i get it
[18:55] <DammitJim> sarnold, are you around today? good afternoon
[18:55] <noname01x3> pastebin came up. now all i have to do is get the text....(still rebooting)
[18:56] <DammitJim> is there a problem with setting systemd-journald to persistent?
[18:56] <sarnold> hey DammitJim :) I wouldn't have thought so, but in that bug report yesterday I got the impression that using it might lead to .. wel,l the exact issue you were facing yesterday
[18:57] <DammitJim> LOL
[18:57] <DammitJim> well, making it persistent has actually somehow helped and the server reboots just fine
[18:57] <sarnold> dude
[18:57] <DammitJim> should I be concerned with leaving it the way it is?
[18:57] <sarnold> just when I think I understand how things work..
[18:57] <noname01x3> Right now I'm at "Welcome to emergency mode! After logging in, type "journalctl -xb" to view system logs, "systemctl" to reboot, "systemctl default" or ^d to try again to boot into default mode. Press Enter for maintenance.
[18:57] <DammitJim> LOL... same thing here... I mean, the log still says that it couldn't unmount /var
[18:58] <DammitJim> dammit, that means that there are more problems, huh?
[18:58] <sarnold> DammitJim: it should be fine to keep it on persistent. your logging fielsystem may see twice the write rate that it had before, since now rsyslog -and- journald are logging, but for most people that really shouldn't be a burden.
[18:58] <DammitJim> ok, thanks
[18:59] <DammitJim> it's sad that one only has a limited amount of time and to get to the root of a problem sometimes one has to just skip it until later
[18:59] <DammitJim> whatever later is...
[18:59] <noname01x3> interesting
[18:59] <JemalMoha>  How to Stop any daemons that is listening on the default SMTP port. (Newbie Here!)
[18:59] <sarnold> DammitJim: yeah. from that bug report it looks sort of like there's no real fix in sight, either
[19:00] <sarnold> Noname01x2: so you're stuck in a rescue mode? o_O oy. that sounds like an annoying thing to sort out.
[19:00] <noname01x3> yeah
[19:00] <noname01x3> it boots into this mode and thats what I was trying to tell you guys. I cant even open any programs
[19:01] <noname01x3> how can I copy and paste the error if I cant even copy n paste
[19:01] <sarnold> aha
[19:01] <noname01x3> I see a red Failed
[19:01] <noname01x3> Failed to start LSB: AppArmor initialization. at the end of the line.
[19:02] <noname01x3> everything else has a green OK
[19:02] <noname01x3> Fix it please. Thanks.
[19:03] <noname01x3> What about the packages? Can we do something with that?
[19:06] <sarnold> noname01x3: what does dmesg | grep -i apparmor report?
[19:07] <JemalMoha> Help ?   How to Stop any daemons that is listening on the default SMTP port. (Newbie Here!)
[19:07] <sarnold> JemalMoha: use netstat -anp | grep :25 to find whatever it listening on port 25, then either use systemctl stop <name> to stop the thing (16.04 lts and newer), or 'stop name' to stop the thing (14.04 lts and older)
[19:09] <JemalMoha> @sarnold thanks!!
[19:10] <noname01x3> sarnold it reports: {       0.001320] AppArmor: AppArmor disabled by boot time parameter
[19:10] <sarnold> noname01x3: hrm, I wouldn't have expected that to cause the boot to fail.
[19:11] <noname01x3> [       29.453556] system[1]: systemd 229 running in system mode.
[19:11] <noname01x3> its all ur fault. u broke it. :-)
[19:11] <noname01x3> fyi i made up that last line.
[19:11] <noname01x3> ..about u braking it...
[19:12] <noname01x3> also it says: (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK...
[19:13] <sarnold> that bit just says that systemd was configured with support for all those different tools
[19:13] <noname01x3> Well the app armor is important.
[19:13] <noname01x3> oh
[19:13] <noname01x3> app armor is in red
[19:13] <sarnold> hehe, yes, I feel the same way, I've been working on apparmor since 2000 :)
[19:14] <noname01x3> :-)
[19:14] <noname01x3> I'm confident you can help me.
[19:17] <sarnold> noname01x3: are there better errors in the logs? /var/log/syslog for example?
[19:17] <noname01x3> how do i get that
[19:18] <sarnold> less /var/log/syslog probably -- there's a chance that the things causing you trouble may not say 'apparmor' directly in the lines..
[19:18] <noname01x3> I would live to give you all possible info.
[19:18] <noname01x3> ok a lot of info cam ein
[19:19] <sarnold> jump to the end and then start scrolling backwrads..
[19:19] <tyler3332> I have an ssh key with a passphrase, but Im not getting asked to put my passphrase in, why is this? Its for a local VM.
[19:19] <sarnold> tyler3332: ssh-agent or a keyring daemon perhaps?
[19:19] <ctjctj> noname01x3, in case you did not know, you can use a slash (/) to start a search in less.  So find your apparmor error, then use page up down to move up and see what was happing near there.  You can use 'G' to go to the end of the file and '?' to search backwards.
[19:19] <tyler3332> how do I check?
[19:20] <ctjctj> tyler3332, ssh-add -L
[19:20] <tyler3332> yup there is a key there
[19:20] <tyler3332> is that ssh-agent?
[19:21] <ctjctj> That means you are connected to an ssh-agent.  When the key is added to the agent you were asked to provide the passphrase.
[19:21] <tyler3332> ah ok
[19:21] <ctjctj> It is assumed that ssh-agent will take good care of your private keys.  I.e. it wont leak them to just any old person.
[19:22] <tyler3332> ok thanks.
[19:22] <ctjctj> ssh-add is how you add (and remove?) keys from the agent.  -l will give you finger prints and -L will give you the public key half.
[19:22] <tyler3332> do keys get added automatically? cause I didnt add it myself
[19:23] <ctjctj> When I'm dealing with clients lacking clues we'll get them to log in with a password verify that they have a working agent with ssh-add -l then have them give us the ssh-add -L output so that we can update "authorized_keys" and turn off password access.
[19:24] <sarnold> definitely good idea
[19:24] <ctjctj> (reality is that we have a special VM with a guest account that the log into to runn ssh-add -L.  That way we never allow password access to our servers to our clients.
[19:24] <sarnold> DUDE
[19:24] <sarnold> that's great :)
[19:24] <noname01x3> sarnold i cant tell what msg is good and whats bad
[19:24] <ctjctj> There is *exactly* one password on our servers.  For root which I know for emergency access from the KVM console.
[19:25] <sarnold> noname01x3: can you copy-and-paste it to paste.ubuntu.com?
[19:25] <noname01x3> I have no idea how.
[19:26] <noname01x3> maybe i should have booted into advanced mode
[19:26] <ctjctj> sarnold, take a look at pam_ssh_agent_auth for how to allow sudo access based on ssh keys.
[19:26] <ctjctj> noname01x3, use pastebinit like I pointed you at.
[19:27] <sarnold> ctjctj: he's stuck in a rescue mode, no networking, no nothing :/
[19:27] <ctjctj> sarnold, service networking start ?
[19:28] <sarnold> worth a try
[19:30] <coreycb> beisner, can you promote nova - 2:15.0.0~b2-0ubuntu4~cloud0 to ocata-proposed please?
[19:31] <noname01x3> i believe i have network access
[19:31] <noname01x3> it says starting raise network interfaces....
[19:31] <sarnold> ctjctj: pam_ssh_agent_auth.. that's odd. it's better than passwords for sudo but I'd be afraid that the ssh key lifetimes useful for logging in to hosts may not match sudo lifetimes very well :)
[19:31] <noname01x3> [OK]Started ifup for eth0.
[19:32] <ctjctj> sarnold, I don't follow.  what do you mean by ssh key lifetimes?
[19:32] <sarnold> ctjctj: well, back in the day I used to use ssh-add -t 300
[19:32] <coreycb> beisner, that should help get puppet CI back in order for mwhahaha
[19:33] <ctjctj> sarnold, ah, we don't  we leave the key in for the duration of the session.
[19:33] <sarnold> ctjctj: but then I started doing tasks on VMs that would take an hour, etc, so I wound up moving to ssh-add -t 3600 ... and then eventually the tasks got long enough that I gave up on -t entirely and just add keys to the agent that last -forever-
[19:33] <ctjctj> yep.
[19:33] <ctjctj> So the life time would be fine for sudo work.
[19:33] <sarnold> but I certainly don't want more than ~five minutes of allowed sudo window
[19:33] <sarnold> not that that's perfect
[19:34] <sarnold> an open terminal is a disaster anyway
[19:34] <sarnold> but still
[19:34] <ctjctj> sarnold, ah.  I understand.  I'm dealing with a bunch of Devs that are Drupal/Wordpress people.  Teaching them to do anything safely is hard.
[19:35] <sarnold> ctjctj: it certainly doesn't come naturally. :)
[19:35] <noname01x3> Maintenance mode
[19:35] <beisner> coreycb, ok, promoted. ye shall haz bridge utils.
[19:35] <coreycb> beisner, awesome
[19:35] <ctjctj> My current favorite is that our "director of creative direction" has been giving out the company user name/password (.htpasswd) to clients instead of making them use their individual passwords)
[19:35] <noname01x3> ctjctj so if I  have network access. then what
[19:36] <sarnold> noname01x3: sweet. apt-get install pastebinit, and then pastebinit /var/log/syslog -- if it looks like something you'd want to share with the world
[19:36] <ctjctj> noname01x3, now you can do things like apt-get install pastebinit and use that to get us bits and bobs of what's going on in your system.
[19:36] <sarnold> ctjctj: ouch. just .. ouch.
[19:37] <noname01x3> reading package
[19:37] <noname01x3> done
[19:37] <ctjctj> sarnold, yeah.  Ouch is the word for this guy.  He manged to make his way onto my s___ list about 6 months ago.  Normally it takes about 2 weeks to work your way off.  He is deeper on the list than when he was placed on it.
[19:38] <sarnold> ctjctj: gotta hand it to him, he does sound -creative-
[19:38] <noname01x3> installed
[19:38] <noname01x3> so now what
[19:38] <noname01x3> im at root
[19:38] <sarnold> pastebinit /var/log/syslog
[19:39] <ctjctj> Friday I get a client to agree to X,Y and Z.  This makes things so much easier for what I'm developing.  Monday he talks to the client and undid all of that client work.  So now the project is in limbo for another two weeks until client reaches a decision, again.
[19:39] <noname01x3> sarnold it says: http://paste.ubuntu.com/23859388/
[19:39] <ctjctj> noname01x3, *yes*!
[19:39] <sarnold> noname01x3: great :) moment..
[19:40] <noname01x3> Congrats to me.
[19:40] <noname01x3> This is fun
[19:40] <noname01x3> My boss will be unhappy if this isn't fixed, however.
[19:40] <ctjctj> noname01x3, sorry about that.  Part of the problem is that we forget to specify everything.
[19:41] <noname01x3> ctjctj ok so thats ok. I get it. You didn't understand how low my skill level was.
[19:41] <noname01x3> VERY LOW. But I'm a smart guy.
[19:41] <ctjctj> sarnold, that looks like a desktop boot.   How dare you?
[19:43] <ctjctj> sarnold, would dmesg hold it?
[19:43] <ctjctj> I'm thinking that systemd might not be running yet.
[19:43] <ctjctj> syslogd
[19:44] <ctjctj> noname01x3, on that server: dmesg| pastebinit
[19:44] <noname01x3> uhh ok. let me see
[19:45] <noname01x3> ctjctj Http://paste.ubuntu.com/23859412/
[19:46] <sarnold> [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.4.0-59-generic.efi.signed root=/dev/mapper/BNY--VS--CA--vg-root ro security=selinux selinux=1
[19:46] <sarnold> no wonder apparmor init script isn't happy :)
[19:46] <ctjctj> noname01x3, translation.  fix your boot command line.
[19:47] <ctjctj> selinux and aptarmor don't play well together (IIRC)
[19:47] <noname01x3> okay. how do we do that
[19:47] <sarnold> indeed :)
[19:47] <sarnold> maybe someday
[19:47] <sarnold> but not this year
[19:47] <sarnold> and probably not next year
[19:47] <noname01x3> Thats ok. We have time.
[19:48] <ctjctj> sarnold, apt-get purge selinux ?
[19:48] <noname01x3> But we need this up and running asap
[19:48] <noname01x3> lol
[19:48] <ctjctj> maybe with an * in there some place?
[19:49] <noname01x3> ctjctj someone doesn't like selinux.
[19:49] <noname01x3> ctjctj yes thank u.
[19:49] <ctjctj> noname01x3, you don't at this point.  Or you can explain to "boss" why the system is down? *GRIN*
[19:49] <noname01x3> :-)
[19:49] <ctjctj> I'm waiting for sarnold to approve that line
[19:49] <sarnold> good question, I don't know selinux packging well enough to know if they automatically add selinux=1 stuff to the kernel command line..
[19:49] <sarnold> apt-get purge 'selinux*' would be a decent starting point anyway
[19:50] <ctjctj> sarnold, I'd start with the purge, check /boot/grub.menu (or whatever it is this week) and if it is still bad we'll update it via grub stuff.
[19:50] <tarpman> careful, that might match libselinux as well
[19:50] <noname01x3> sarnold will it give me a warning if it will do something i dont like?
[19:50] <sarnold> noname01x3: not really
[19:50] <sarnold> linux is like that
[19:50] <ctjctj> noname01x3, nope.  It's unix.    I ended up in the source code to find out that -y means --blocksize...
[19:53] <coreycb> zul, that should unblock some things
[19:53] <zul> coreycb: i saw
[19:53] <zul> coreycb: oslo.middleware
[19:53] <ikonia> -y means blocksize ???
[19:53] <ikonia> who thought that out
[19:53] <noname01x3> I have a backup checkpoint from this morning.
[19:53] <noname01x3> I can put it right back if it goes bad
[19:53] <ctjctj> ikonia, yeah.
[19:53] <noname01x3> or maybe I dont know what im talking about.
[19:54] <sarnold> ctjctj: eww!
[19:54] <ctjctj> noname01x3, I'm glad you have that checkpoint.  Means we don't have to worry about xxxbadcommamndXXX -rf / mean people.
[19:56] <noname01x3> ctjctj, thanks. Yeah I believe I can try it with no worries. And my boss is constantly asking me whats new and I would love to have an answer, other than repeating what you guys are saying, which causes him to ask "And".
[19:57] <ctjctj> noname01x3, what version of ubuntu are you running? 14.04LTS?
[19:58] <sarnold> probably 16.04 lts, with kernel version 4.4.x
[19:58] <ctjctj> sarnold, ok.
[19:58] <noname01x3> 16.04 LTS
[19:59] <ctjctj> noname01x3, once you have the purge done: pastebinit /boot/grub/grub.cfg
[19:59] <noname01x3> ctjctj, ok i will do the purge now.
[20:00] <noname01x3> apt-get purge selinux
[20:00] <noname01x3> Done
[20:00] <noname01x3> waait it says click y to continue
[20:01] <ctjctj> noname01x3, wrong command.
[20:01] <ctjctj> apt-get purge 'selinux*'
[20:01] <noname01x3> pressed y
[20:01] <ctjctj> That's ok.  When you are done run the second purge too.
[20:01] <ctjctj> sarnold, updated my original command.
[20:02] <ctjctj> if it says anything about "grub" as it is doing that purge let us know please.
[20:02] <sarnold> good idea, apt-get purge 'selinux*' looks better
[20:02] <ctjctj> sarnold, well I did cut and paste your version... *GRIN*
[20:03] <sarnold> hehe
[20:03] <noname01x3> ctjctj, http://paste.ubuntu.com/23859484
[20:04] <noname01x3> whats the second purge?
[20:04] <sarnold> the second purge would remove any selinux policies that might have been installed via tha packages
[20:04] <ctjctj> it adds a wildcard to search for other packages named selinux[anything]
[20:04] <noname01x3> ohh ok lets not lol
[20:04] <ctjctj> So if there is a selinux-break-nonames-system the * would match -break-nonames-system
[20:05] <ctjctj> You want to delete those other things noname01x3.
[20:05] <noname01x3> ctjctj, unless you insist
[20:05] <ctjctj> sarnold, looks like the security=selinux and selinux=1 are still in the grub.cfg
[20:05] <sarnold> "security=selinux selinux=1
[20:05] <sarnold> yes that :) heh
[20:05] <noname01x3> ahhhhhhhhh
[20:05] <noname01x3> lets get it out!!!!!!
[20:06] <noname01x3> ok whats the purge. lets do it
[20:06] <ctjctj> noname01x3, what's happening is that sarnold and I could very easily tell you how to remove that part in grub.cfg and update grub to have your system boot.  But my fear is that the next time you got a kernel update the selinux issue would come back.
[20:06] <ctjctj> apt-get purge 'selinux*'
[20:07] <ctjctj> noname01x3, pastebinit /etc/default/grub.cfg
[20:07] <ctjctj> noname01x3, do you speak vim, emacs or ed/ex?
[20:10] <Noname01x2> what the???
[20:10] <Noname01x2> i was disconnected
[20:11] <ctjctj> Noname01x2, pastebin /etc/default/grub.cfg after the purge
[20:11] <ctjctj> Noname01x2, do you speak vim, emacs, ed/ex, or nano?
[20:11] <noname01x3> im back
[20:11] <noname01x3> ctjctj, none of the above
[20:11] <ctjctj> noname01x3, pastebinit /etc/default/grub.cfg
[20:12] <ctjctj> if that purge completed.
[20:12] <noname01x3> unable to read from /etc/default/grub.cfg
[20:13] <ctjctj> sorry /etc/default/grub
[20:13] <ctjctj> to many windows with some being 14.04 and some 16.10 with slightly different naming conventions.
[20:14] <noname01x3> ctjctj, http://paste.ubuntu.com/23859650/
[20:15] <ctjctj> noname01x3, nano /etc/default/grub  go to the line that says "GRUB_CMDLINE_LINUX_DEFAULT" and remove everything between the quotes.
[20:15] <ctjctj> nano is designed for my 6yo.  So I'm hoping you can drive it... *GRIN*
[20:16] <sarnold> that's the first package I purge every install :)
[20:16] <ctjctj> sarnold, my IDE is emacs.... *GRIN*
[20:17] <noname01x3> ctjctj, I'm doing what you told me. Stand by...
[20:17] <sarnold> ctjctj: do you also use it for irc, email, web, etc? :)
[20:18] <ctjctj> sarnold, is there any other way? *grin* Nope.  just my programming editor.  I'm equally at home with vim, vi, ed, ex, jove and a few others.
[20:19] <sarnold> ctjctj: no kidding? I started with vi too long enough, and never figured out how to do more than quit emacs..
[20:19] <ctjctj> though I have used it for email, irc and uunet news
[20:20] <ctjctj> sarnold, I started with a home grown editor that I wrote to interface with a distribute console editing system. (scredit at MSU).  But I was playing "hack" or "nethack" and my fingers learned the vi key movements.
[20:20] <sarnold> and once vi was in my fingers, nothing else quite fits right
[20:21] <ctjctj> about 1989 I was working with Mike Muuse (author of ping and a bunch of other great things) and watched him drive "jove" (jonathans own version of emacs).  Jove was light weight enough to load in a reasonable amount of time vs emacs.  And seeing some of the things  he was doing, multiple windows into the same file, two files open at the same time, cut and paste from one window to another.  and all of it in a 24x80 green
[20:21] <ctjctj> screen terminal.
[20:22] <noname01x3> ctjctj, do i remove the ""?
[20:22] <ctjctj> I switched to jove the next day.  Used jove until emacs for programming and vim for "quick edits"
[20:22] <ctjctj> noname01x3, no.  Leave the ""
[20:22] <ctjctj> sarnold, I have one of my logical units back.  Thanks for the pointer to tgt.
[20:23] <sarnold> ctjctj: ahhh that makes sense. yeah. emacs was way too heavy to load on the shared system I started with, I didn't want the other users upset :)
[20:23] <sarnold> ctjctj: sweet! how's the transition?
[20:23] <noname01x3> ctjctj, im done.
[20:24] <ctjctj> the transition is a pita.  It is all command line controlled.  I haven't located a configuration file yet so about 6 long command lines to define the target, define the LUN, define the users, attach the users, open the target to the initiator.
[20:24] <noname01x3> I think I want to become an ubuntu expert now.
[20:24] <ctjctj> noname01x3, update-grub
[20:25] <noname01x3> ctjctj, im still in nano.
[20:25] <noname01x3> ctjctj, do i do exit?
[20:25] <ctjctj> isn't it in the menu at the bottom? (ctrl-x)
[20:25] <ctjctj> it will ask you to save the file and such.   I get it. Its very easy to use. I just like to follow instructions to a tee.
[20:26] <noname01x3> ctjctj, ok now we do that other thing yo usaid....
[20:27] <noname01x3> updating...
[20:27] <ctjctj> sarnold, as far as I can tell this is a cleaner interface.  iscsitarget has a command line interface but it does not save any dynamic modifications.  So you have to modify the target/lun dynamicly, update the configuration file.  Hope you got it right.  If you restart iscsitarget your initatiors hate you.
[20:27] <ctjctj> sarnold, this is all dynamic which is good in that way.
[20:27] <noname01x3> ctjctj, done.
[20:28] <ctjctj> noname01x3, pastebinit /boot/grub/grub.cfg
[20:28] <noname01x3> ctjctj, http://paste.ubuntu.com/23859711/
[20:29] <ctjctj> noname01x3, what is your favorite God?
[20:29] <ctjctj> noname01x3, cross fingers, start praying and type "shutdown -r +1"
[20:32] <sarnold> ctjctj: owww, I never really thought about trying to change configs of these things on the fly
[20:32] <sarnold> ctjctj: (i've really only got the one good computer, so even though I've read all the docs, once upon atime, I never needed to use the things)
[20:34] <noname01x3> sorry. disconnected
[20:34] <noname01x3> ctjctj, what now
[20:34] <ctjctj> sarnold, I'm sitting in my home office with primary box 16.10, two vm's 14.04lts, two other desk tops for wives and kids computer.  And then a dozen primary servers and two dozen dev VM sandboxes running 14.04lts.  I've broken everything atleast once.
[20:34] <ctjctj> noname01x3, cross fingers, start praying, and type "shutdown -r +1"
[20:34] <ctjctj> and stop disconnecting. *grin*
[20:35] <sarnold> ctjctj: hehe, breaking things, a good way to learn
[20:35] <ctjctj> sarnold, changing /etc/default files is new to you?
[20:36] <sarnold> ctjctj: yeah. I felt more at home with editing the one big /etc/rc with the ifconfig and route commands and what not just stuffed in there.
[20:36] <sarnold> sysvrc was a stretch, but I got there.
[20:36] <sarnold> /etc/default/ .. newfangled things. I never think to look there first, only when reading the sources eventually shows the ". /etc/default/whatever" in the script I'm reading
[20:36] <ctjctj> sarnold, I was once at the National Cancer Research center in Fredricks MD (I think I remember correctly).  I had 48 hours to install unicos on their cray X/MP.  Documentation says it takes two weeks.  It took me 32 hours.  Having the director pop in ever 2 hours to check on things was stressful.
[20:37] <ctjctj> On the other hand the fact that pizza and Coke-a-cola were always fresh was nice.
[20:37] <noname01x3> ctjctj, sarnold, if this works...
[20:38] <sarnold> ctjctj: daaaang. things were different in those days. linux was a breath of fresh air compared to those cranky old unix machines.
[20:38] <sarnold> ctjctj: my first job had an sco unix thing. no compiler. no header files. but we did have perl..
[20:38] <noname01x3> a start job is running...
[20:38] <ctjctj> sarnold, not really.  Early linux was the pits compared to BSD 4.4 though I hated solaris.  SunOS 4.3+ was a joy to work with.
[20:38] <noname01x3> hmmm.
[20:39] <noname01x3> "OK" AppArmor initialization.
[20:39] <ctjctj> sarnold, I once had to get a CC compiler on to a "binary only" box in order to compile GCC
[20:40] <sarnold> ctjctj: I dunno. I recall my first slackware install much more fondly than the terrible old sco box... heh.
[20:40] <sarnold> ctjctj: the bsdi system I only ever used as a user. it just worked. someone else dealt with everything :)
[20:40] <sarnold> ctjctj: boostrapping gcc though. ugh.
[20:40] <ctjctj> SCO is not a fun version of Linux.
[20:40] <noname01x3> ctjctj, sarnold, its all showing OK so why is it still in emergency mode?
[20:40] <sarnold> ctjctj: rofl
[20:40] <ctjctj> noname01x3, no idea.  Why did you tell it to boot to emergency mode?
[20:40] <noname01x3> uhhh
[20:40] <sarnold> noname01x3: why is it in emergency mode this time?
[20:41] <noname01x3> dont know
[20:41] <ctjctj> (or you can show us syslog and let us try and figure it out from data instead of guessing?)
[20:42] <ctjctj> sarnold, BSDI is not the same as BSD.  BSDI is based on BSD 4.4 open release.  Which is what set Unix free.  Thank you U of California.
[20:42]  * ctjctj needs a control dial for snark level.
[20:43] <sarnold> ctjctj: yeah, I didn't actually use a bsd 4.4. system. just the bsdi derivative. (Thanks microsoft!) and then a freebsd system somewhere along the way, for a month, just for kicks.
[20:43] <ctjctj> I have a few contributions to FreeBSD.
[20:44] <ctjctj> But that was a long long time ago.
[20:44] <noname01x4> wow i came back automatically this time
[20:45] <ctjctj> noname01x4, I told you to stop disconnecting.  How rude.  Makes it hard to help you.... *GRIN*.  We invite you to show us data so we can figure out how you asked your system to boot into emergency mode.
[20:45] <ctjctj> (that's a quiz)
[20:46] <noname01x4> http://paste.ubuntu.com/23859756/
[20:46] <noname01x4> what the
[20:46] <noname01x4> how can I quit
[20:46] <ctjctj> your name changed.
[20:46] <sarnold> ping timeouts are funny things
[20:46] <noname01x4> Im on wifi so maybe thats why
[20:47] <noname01x4> or pint timeouts
[20:47] <sarnold> your irc client didn't respond to the server's PING? request in time, so the server disconnects you
[20:47] <noname01x4> okj
[20:47] <ctjctj> sarnold, that's exactly the same syslog as he posted last time.
[20:47] <sarnold> normally your client will notice that it hasn't had a PING? request in a while and disconnect/reconnect on its own. and your name would still be in use.
[20:48] <sarnold> well now that's odd. I thought those timestamps didn't look right..
[20:48] <noname01x4> so what now
[20:48] <ctjctj> noname01x4, I'm thinking.
[20:48] <ctjctj> sarnold, he can write to his disk.  So the disk is not read-only.
[20:49] <ctjctj> noname01x4, do: ls -l /var/log/syslog and take note of the size and date
[20:49] <ctjctj> noname01x4, then service rsyslog start
[20:49] <ctjctj> check /var/log/syslog and see if anything changed.
[20:55] <noname01x4> ctjctj, ok
[20:55] <noname01x4> ctjctj, its doing something....
[20:56] <ctjctj> syslog is growing or the date changed?
[20:56] <noname01x4> nothing is happening
[20:56] <noname01x4> just sitting there.
[20:58] <ctjctj> noname01x4, did you run ls -l /var/log/syslog again?  Did the values change?
[20:58] <noname01x4> ctjctj, no they did not change.
[20:58] <sarnold> o_O
[20:58] <noname01x4> 13:19
[20:59] <noname01x4> wow u made the guy quit
[20:59] <ctjctj> did it tell you why it was entering emergency mode?
[20:59] <noname01x4> nope
[20:59] <ctjctj> Did you see any "fails" on the boot?
[20:59] <noname01x4> i just typed reboot
[20:59] <ctjctj> noname01x4, it is almost always better to use "shutdown -r +1" than a simple "reboot"
[21:00] <noname01x4> shutdown -r +1?
[21:00] <noname01x4> what is that
[21:00] <ctjctj> yes.  Like I said up above.
[21:00] <ctjctj> try "man shutdown"
[21:01] <noname01x4> ok so far it says "mounted /boot/efi.
[21:01] <noname01x4> ok
[21:01] <noname01x4> I will use that in the future.
[21:02] <noname01x4> if possible
[21:02] <noname01x4> i tried it and it didnt take, but maybe i typed it wrong
[21:02] <ctjctj> you should be able to type "man command" for any command we give you.  If you don't know what the command does or the options we are giving you, you should run man to see what it is you are being told to do.
[21:03] <noname01x4> its dpoing something
[21:03] <noname01x4> qq
[21:03] <ctjctj> Which doesn't help when people are intentionally attempting to mess with you, the old: Your documents don't look right.  You need to make them look better.  Open up a DOS window and type "format c:" that will format all your documents to C things better
[21:04] <noname01x4> interesting
[21:04] <noname01x4> I just cant get this boot going
[21:04] <noname01x4> always emergency mode
[21:05] <ctjctj> Of course *I* never did that.  I just sent out CD's with Linux or FreeBSD labeled "Windows '95 update'
[21:05] <noname01x4> cant we just change somethjing in the boot to say startttt normally
[21:05] <ctjctj> what does journalctl -xb tell you?
[21:06] <noname01x4> so much info
[21:06] <ctjctj> Ok.
[21:06] <ctjctj> start the network then
[21:06] <noname01x4> nothing looked bad except maybe one or two things
[21:06] <ctjctj> journalctl -l -xb | pastebinit
[21:06] <sarnold> off to lunch, back eventually, good luck guys :)
[21:06] <ctjctj> sarnold, thanks for the discussions.
[21:07] <sarnold> thanks to you, too, very helpful :D
[21:08] <noname01x4> sarnold, you are fantastic. enjoy your meal.
[21:08] <noname01x4> ctjctj, http://paste.ubuntu.com/23859865
[21:10] <ctjctj> noname01x4, did you see anything in RED as that scrolled by?
[21:11] <noname01x4> Yeah a few thinks., ctjctj.
[21:11] <noname01x4> pci had a problem
[21:12] <noname01x4> but who needs a pci
[21:12] <ctjctj> Ok.  how many disk drives do you have attached to this machine?
[21:12] <ctjctj> pastebinit /etc/fstab
[21:14] <ctjctj> It is looking bad for a disk drive.  I would also like to see "blkid | pastebinit"
[21:14] <noname01x4> pci: fatal?
[21:15] <ctjctj> According to your journal you failed to mount /data because /dev/sdc1 does not seem to exist.  I'm exploring that right now.  So if you can get me those two pastebin's it would be helpful.
[21:15] <noname01x4> ctjctj, http://paste.ubuntu.com/23859904
[21:15] <noname01x4> there was a third hard drive thats not there
[21:16] <ctjctj> noname01x4, *blinks* what do you mean there is suppose to be a third drive that is "not there"?
[21:16] <ctjctj> Did you take the drive out of the box?
[21:17] <noname01x4> ca boto and ca data
[21:17] <ctjctj> noname01x4, I don't follow.
[21:17] <noname01x4> cs data
[21:17] <noname01x4> wait
[21:17] <noname01x4> im typoing
[21:17] <noname01x4> wait my boss is telling me something about hard drives.
[21:18] <noname01x4> he is not sure what he's talking about
[21:18] <noname01x4> he says there should be a drive called data
[21:18] <noname01x4> ok nvm
[21:18] <ctjctj> Yes, there should be a drive that is mounted on /data
[21:18] <ctjctj> And I asked for "blkid | pastebinit" so I can see what drives you have
[21:19] <noname01x4> ctjctj, http://paste.ubuntu.com/23859927/
[21:20] <ctjctj> Ok.
[21:20] <noname01x4> yeah I would hope not. uhggg. now hes on the phone.
[21:23] <noname01x4> timed out waiting for device dev-sdc1.device
[21:23] <noname01x4> that may be normal i guess
[21:24] <ctjctj> noname01x4, There is a sda and a sdb drive that shows in your dmesg boot sequence.  There is no reference to sdc.
[21:24] <ctjctj> Your fstab says to mount /dev/sdc1 as /data.  If that fails then bad things happen.  Such as ending up in emergency mode.
[21:24] <noname01x4> ohh
[21:24] <ctjctj> So when I hear the answer to the question "Did you remove a drive from the box" I can move forward on debugging your issue.
[21:25] <noname01x4> ok stand by
[21:25] <ctjctj> This is one of those times where people get slapped with a dead fish "My system doesn't boot"  four hours of debugging later "Do you think that removing this extra card from the computer might have caused and issue?"
[21:26] <ctjctj> noname01x4, is this a bare metal box or is it virtualized?
[21:26] <ctjctj> And yes, my snark level is high.
[21:28] <noname01x4> virtualized
[21:28] <ctjctj> Are all three drives provided as virtualized drives or is one of them provided as an iscsi target?
[21:29] <noname01x4> that is an interesting question. give me a few more minutes to get the answers.
[21:43] <ctjctj> noname01x4, any answers yet?
[21:44] <ctjctj> noname01x4, I'm out of here in 16.25 minutes for an hour.
[22:01] <ctjctj> AFK
[22:09] <noname01x4> hey hey
[22:09] <noname01x4> ctjctj, my boss is saying there was a third physical drive that was removed
[22:09] <noname01x4> ctjctj, but there error was there before the drive was removed.
[22:14] <noname01x3> ok
[22:15] <noname01x3> ctjctj, all the drives are virtual
[22:15] <noname01x3> ctjctj, and the error happed before the third physical drive was removed.
[22:28] <Noname01x2> crazy
[22:29] <Noname01x2> my boss told me to continue working on this until its fixed
[23:27] <jelly> "flogging shall continue until morale is improved"
[23:50] <ctjctj> noname01x3, are you here?