service_status_check task generates exceptions when no data found in ES index

When run start_dev_env.sh, there is some errors related to celery tasks.

Trying https://github.com/mobz/elasticsearch-head/archive/master.zip...
Downloading .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Installed mobz/elasticsearch-head into /usr/share/elasticsearch/plugins/head
-> Installing royrusso/elasticsearch-HQ...
Trying https://github.com/royrusso/elasticsearch-HQ/archive/master.zip...
Downloading ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Operations to perform:
Synchronize unmigrated apps: django_admin_bootstrapped, drfes, staticfiles, messages, djoser, rest_framework_swagger, django_extensions, accounts, rest_framework, polymorphic
Apply all migrations: authtoken, core, sessions, admin, auth, contenttypes, user, tenants
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states...............................................................................DONE
DONE
Applying tenants.0001_initial... OK
Applying contenttypes.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial...Installed royrusso/elasticsearch-HQ into /usr/share/elasticsearch/plugins/HQ
OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying user.0001_initial... OK
Applying admin.0001_initial... OK
Applying authtoken.0001_initial... ---> f8a54c98b018
OK
Applying core.0001_initial...Removing intermediate container b2a4014fbd83
Step 4 : ENV ES_CLUSTERNAME "goldstone" ES_HEAP_SIZE "1g"
---> Running in 088e8520328e
---> 5ddff65eb6e2
Removing intermediate container 088e8520328e
Step 5 : ENV ES_JAVA_OPTS "      -Des.cluster.name=goldstone       -Des.node.name=${HOSTNAME}       -Des.index.search.slowlog.threshold.query.warn=10s       -Des.index.search.slowlog.threshold.query.info=5s       -Des.index.search.slowlog.threshold.query.debug=2s       -Des.index.search.slowlog.threshold.query.trace=500ms       -Des.index.search.slowlog.threshold.fetch.warn=1s       -Des.index.search.slowlog.threshold.fetch.info=800ms       -Des.index.search.slowlog.threshold.fetch.debug=500ms       -Des.index.search.slowlog.threshold.fetch.trace=200ms       -Des.index.indexing.slowlog.threshold.index.warn=10s       -Des.index.indexing.slowlog.threshold.index.info=5s       -Des.index.indexing.slowlog.threshold.index.debug=2s       -Des.index.indexing.slowlog.threshold.index.trace=500ms       -Des.monitor.jvm.gc.young.warn=1000ms       -Des.monitor.jvm.gc.young.info=700ms       -Des.monitor.jvm.gc.young.debug=400ms       -Des.monitor.jvm.gc.old.warn=10s       -Des.monitor.jvm.gc.old.info=5s       -Des.monitor.jvm.gc.old.debug=2s       "
---> Running in 3a9daa409355
---> 453d1df54285
Removing intermediate container 3a9daa409355
Step 6 : COPY config /usr/share/elasticsearch/config
---> 87fb226c1200
Removing intermediate container 310ee6d9c6ae
Step 7 : EXPOSE 9200 9300
---> Running in 55e8a7693d8d
---> 23a361a8c776
Removing intermediate container 55e8a7693d8d
Successfully built 23a361a8c776
Creating goldstoneserver_gssearch_1
OK
Applying sessions.0001_initial... OK
Building gstaskq
Step 1 : FROM solinea/redis:3.0.4
Installed 23 object(s) from 1 fixture(s)
0 static files copied to '/home/jor/devel/goldstone-server/docker/goldstone-web/static', 306 unmodified.
3.0.4: Pulling from solinea/redis
f8efbffe7b95: Already exists
a3ed95caeb02: Pulling fs layer
ea5dbfb1853f: Pulling fs layer
3d687babc696: Waiting
0a4af429773e: Waiting
9aec05558ecc: Waiting
c8fc964dd59c: Waiting
f27c107136a9: Waiting
starting postinstall steps for docker environment
Setting up Django admin account.
Creating Django admin account.
done.
  ''-._           Redis 3.0.4 (00000000/0) 64 bit
gskibana_1 | {"name":"Kibana","hostname":"449cd53dddb7","pid":17,"level":30,"msg":"Listening on <IP_ADDRESS>:5601","time":"2016-05-05T14:46:55.220Z","v":0}
gssearch_1 | [2016-05-05 14:43:28,581][INFO ][plugins                  ] [07bd8c419a72] loaded [], sites [head, HQ]
gstaskq_1  |   .- .-```.  ```\/    _.,_ ''-._                                    gsdb_1     | The default database encoding has accordingly been set to "UTF8". gstaskq_1  |  (    '      ,       .-`  | `,    )     Running in standalone mode gssearch_1 | [2016-05-05 14:43:28,653][INFO ][env                      ] [07bd8c419a72] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/disk/by-uuid/5817c876-5526-49fe-9b01-3a9a7447b398)]], net usable_space [185.5gb], net total_space [234.5gb], types [ext4] gsdb_1     | The default text search configuration will be set to "english". gstaskq_1  |  |`-._`-...-` __...-.-.|'_.-'|     Port: 6379 gsdb_1     |  gstaskq_1  |  |   -. ._    /     _.-'    |     PID: 1 gstaskq_1  |   -._    -._  -./  .-'    .-'
gstaskq_1  |  |-._-. -.__.-'    _.-'_.-'|                                   gstaskq_1  |  |    -.-._        _.-'_.-'    |           http://redis.io         gstaskq_1  |   -._    -._-..-'.-'    .-'
gstaskq_1  |  |-._-. -.__.-'    _.-'_.-'|                                   gssearch_1 | [2016-05-05 14:43:31,794][INFO ][node                     ] [07bd8c419a72] initialized gsdb_1     | Data page checksums are disabled. gstaskq_1  |  |    -.-._        _.-'_.-'    |                                   gsdb_1     |  gssearch_1 | [2016-05-05 14:43:31,794][INFO ][node                     ] [07bd8c419a72] starting ... gstaskq_1  |   -._    -._-..-'_.-'    _.-'
gsdb_1     | fixing permissions on existing directory /var/lib/postgresql/data ... ok
gstaskq_1  |       -._    -..-'    _.-'
gstaskq_1  |           -._        _.-'                                            gstaskq_1  |               -..-'
gstaskq_1  |
gstaskq_1  | 1:M 05 May 14:43:56.140 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
gstaskq_1  | 1:M 05 May 14:43:56.140 # Server started, Redis version 3.0.4
gstaskq_1  | 1:M 05 May 14:43:56.140 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
gstaskq_1  | 1:M 05 May 14:43:56.140 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
gstaskq_1  | 1:M 05 May 14:43:56.140 * The server is now ready to accept connections on port 6379
gssearch_1 | [2016-05-05 14:43:31,917][INFO ][transport                ] [07bd8c419a72] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/<IP_ADDRESS>:9300]}
gssearch_1 | [2016-05-05 14:43:31,995][INFO ][discovery                ] [07bd8c419a72] goldstone/x1CQzsYXRjiouuaG60qgRQ
gssearch_1 | [2016-05-05 14:43:35,793][INFO ][cluster.service          ] [07bd8c419a72] new_master [07bd8c419a72][x1CQzsYXRjiouuaG60qgRQ][07bd8c419a72][inet[/<IP_ADDRESS>:9300]], reason: zen-disco-join (elected_as_master)
gssearch_1 | [2016-05-05 14:43:35,832][INFO ][http                     ] [07bd8c419a72] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/<IP_ADDRESS>:9200]}
gssearch_1 | [2016-05-05 14:43:35,832][INFO ][node                     ] [07bd8c419a72] started
gssearch_1 | [2016-05-05 14:43:36,062][INFO ][gateway                  ] [07bd8c419a72] recovered [0] indices into cluster_state
gssearch_1 | [2016-05-05 14:46:55,144][DEBUG][action.admin.cluster.health] [07bd8c419a72] observer: timeout notification from cluster service. timeout setting [5s], time since start [5s]
gssearch_1 | [2016-05-05 14:51:55,001][INFO ][cluster.metadata         ] [07bd8c419a72] [logstash-2016.05.05] creating index, cause [auto(bulk api)], templates [logstash], shards [1]/[0], mappings [syslog, default]
gssearch_1 | [2016-05-05 14:51:56,063][INFO ][cluster.metadata         ] [07bd8c419a72] [logstash-2016.05.05] update_mapping [syslog] (dynamic)
gssearch_1 | [2016-05-05 14:51:58,121][INFO ][cluster.metadata         ] [07bd8c419a72] [logstash-2016.05.05] update_mapping [syslog] (dynamic)
gssearch_1 | [2016-05-05 14:52:01,024][TRACE][index.indexing.slowlog.index] [07bd8c419a72] [logstash-2016.05.05][0] took[539.2ms], took_millis[539], type[syslog], id[AVSBZ3BCX5tKSO6UrKRF], routing[], source[{"@timestamp":"2016-05-05T14:39:00.619Z","message":"<179>2016-05-05T14:39:00.619448+00:00 rdo-kilo journal: 2016-05-05 14:39:00.616 4066 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on <IP_ADDRESS>:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.","@version":"1","host":"rdo-kilo","port":32907,"type":"syslog","tags":["filter_16_fail","filter_20_pass","filter_33_fail","openstack_syslog","filter_34_pass"],"syslog5424_pri":"179","syslog_ts":"2016-05-05T14:39:00.619448+00:00","syslog5424_host":"rdo-kilo","log_message":"journal: 2016-05-05 14:39:00.616 4066 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on <IP_ADDRESS>:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.","received_at":"2016-05-05T14:52:00.236Z","syslog_severity_code":3,"syslog_facility_code":22,"syslog_facility":"local6","syslog_severity":"ERROR","event_type":"OpenStackSyslogError","timestamp":"2016-05-05 14:39:00.616","pid":4066,"loglevel":"ERROR","program":"oslo_messaging._drivers.impl_rabbit","request_id_list":"[-]","openstack_message":"AMQP server on <IP_ADDRESS>:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.","component":"keystone"}]
gsdb_1     | creating subdirectories ... ok
gsdb_1     | selecting default max_connections ... 100
gsdb_1     | selecting default shared_buffers ... 128MB
gsdb_1     | selecting dynamic shared memory implementation ... posix
gsdb_1     | creating configuration files ... ok
gsdb_1     | creating template1 database in /var/lib/postgresql/data/base/1 ... ok
gsdb_1     | initializing pg_authid ... ok
gsdb_1     | initializing dependencies ... ok
gsdb_1     | creating system views ... ok
gsdb_1     | loading system objects' descriptions ... ok
gsdb_1     | creating collations ... ok
gsdb_1     | creating conversions ... ok
gsdb_1     | creating dictionaries ... ok
gsdb_1     | setting privileges on built-in objects ... ok
gsdb_1     | creating information schema ... ok
gsdb_1     | loading PL/pgSQL server-side language ... ok
gsdb_1     | vacuuming database template1 ... ok
gsdb_1     | copying template1 to template0 ... ok
gsdb_1     | copying template1 to postgres ... ok
gsdb_1     | syncing data to disk ...
gsdb_1     | WARNING: enabling "trust" authentication for local connections
gsdb_1     | You can change this by editing pg_hba.conf or using the option -A, or
gsdb_1     | --auth-local and --auth-host, the next time you run initdb.
gsdb_1     | ok
gsdb_1     |
gsdb_1     | Success. You can now start the database server using:
gsdb_1     |
gsdb_1     |     postgres -D /var/lib/postgresql/data
gsdb_1     | or
gsdb_1     |     pg_ctl -D /var/lib/postgresql/data -l logfile start
gsdb_1     |
gsdb_1     | waiting for server to start....LOG:  database system was shut down at 2016-05-05 14:41:06 UTC
gsdb_1     | LOG:  MultiXact member wraparound protections are now enabled
gsdb_1     | LOG:  database system is ready to accept connections
gsdb_1     | LOG:  autovacuum launcher started
gsdb_1     |  done
gsdb_1     | server started
gsdb_1     | ALTER ROLE
gsdb_1     |
gsdb_1     |
gsdb_1     | /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init-goldstone-db.sh
gsdb_1     | CREATE ROLE
gsdb_1     | CREATE DATABASE
gsdb_1     | GRANT
gsdb_1     | ALTER ROLE
gsdb_1     | ALTER ROLE
gsdb_1     | ALTER ROLE
gsdb_1     | updating pg_hba.conf
gsdb_1     | LOG:  received SIGHUP, reloading configuration files
gsdb_1     | server signaled
gsdb_1     |
gsdb_1     | LOG:  received fast shutdown request
gsdb_1     | LOG:  aborting any active transactions
gsdb_1     | waiting for server to shut down....LOG:  autovacuum launcher shutting down
gsdb_1     | LOG:  shutting down
gsdb_1     | LOG:  database system is shut down
gsdb_1     |  done
gsdb_1     | server stopped
gsdb_1     |
gsdb_1     | PostgreSQL init process complete; ready for start up.
gsdb_1     |
gsdb_1     | LOG:  database system was shut down at 2016-05-05 14:41:11 UTC
gsdb_1     | LOG:  MultiXact member wraparound protections are now enabled
gsdb_1     | LOG:  database system is ready to accept connections
gsdb_1     | LOG:  autovacuum launcher started
gsdb_1     | LOG:  incomplete startup packet
gsdb_1     | LOG:  incomplete startup packet
gstask_1   |
gstask_1   |  -------------- celery@e58bcb839ad2 v3.1.19 (Cipater)
gstask_1   | ---- **** -----
gstask_1   | --- * ***  * -- Linux-3.16.0-38-generic-x86_64-with-debian-8.2
gstask_1   | -- * - **** ---
gstask_1   | - ** ---------- [config]
gstask_1   | - ** ---------- .> app:         goldstone:0x7fda20fa2b50
gstask_1   | - ** ---------- .> transport:   redis://gstaskq:6379/0
gstask_1   | - ** ---------- .> results:     redis://gstaskq:6379/0
gstask_1   | - *** --- * --- .> concurrency: 4 (prefork)
gstask_1   | -- ******* ----
gstask_1   | --- ***** ----- [queues]
gstask_1   |  -------------- .> default          exchange=default(direct) key=default
gstask_1   |
gstask_1   |
gstask_1   | [tasks]
gstask_1   |   . goldstone.core.tasks.expire_auth_tokens
gstask_1   |   . goldstone.core.tasks.process_alerts
gstask_1   |   . goldstone.core.tasks.prune_es_indices
gstask_1   |   . goldstone.core.tasks.service_status_check
gstask_1   |
gstask_1   | [2016-05-05 14:56:39,225: INFO/MainProcess] Connected to redis://gstaskq:6379/0
gstask_1   | [2016-05-05 14:56:39,234: INFO/MainProcess] mingle: searching for neighbors
gstask_1   | [2016-05-05 14:56:39,268: INFO/Beat] beat: Starting...
gstask_1   | [2016-05-05 14:56:40,243: INFO/MainProcess] mingle: all alone
gstask_1   | /venv/local/lib/python2.7/site-packages/celery/fixups/django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
gstask_1   |   warnings.warn('Using settings.DEBUG leads to a memory leak, never '
gstask_1   |
gstask_1   | [2016-05-05 14:56:40,251: WARNING/MainProcess] /venv/local/lib/python2.7/site-packages/celery/fixups/django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
gstask_1   |   warnings.warn('Using settings.DEBUG leads to a memory leak, never '
gstask_1   |
gstask_1   | [2016-05-05 14:56:40,252: WARNING/MainProcess] celery@e58bcb839ad2 ready.
gstask_1   | [2016-05-05 14:57:00,006: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 14:57:00,010: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[7d112a39-8089-4e0a-ad2e-86b78bc163fd]
gstask_1   | 2016-05-05T14:57:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 14:57:00,011: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 14:57:00,022: INFO/Worker-3] Starting new HTTP connection (1): gssearch
gstask_1   | [2016-05-05 14:57:00,030: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 14:57:00,031: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[7d112a39-8089-4e0a-ad2e-86b78bc163fd] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 14:57:00,032: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[f589ba07-575c-4827-b3ad-1714160c1d33]
gstask_1   | [2016-05-05 14:57:00,046: INFO/Worker-4] Starting new HTTP connection (1): gssearch
gstask_1   | [2016-05-05 14:57:00,140: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[f589ba07-575c-4827-b3ad-1714160c1d33] succeeded in 0.107236084s: None
gssearch_1 | [2016-05-05 14:57:00,803][INFO ][cluster.metadata         ] [07bd8c419a72] [internal-2016.05.05] creating index, cause [auto(bulk api)], templates [], shards [1]/[0], mappings [internal]
gssearch_1 | [2016-05-05 14:57:01,060][INFO ][cluster.metadata         ] [07bd8c419a72] [internal-2016.05.05] update_mapping [internal] (dynamic)
gstask_1   | [2016-05-05 14:58:00,042: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 14:58:00,043: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 14:58:00,044: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[307ef8ab-e6f1-436a-ad78-cd9cd2935b95]
gstask_1   | 2016-05-05T14:58:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 14:58:00,045: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 14:58:00,047: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[818cc652-e104-4426-850a-e771dddea37a]
gstask_1   | [2016-05-05 14:58:00,056: INFO/Worker-2] Starting new HTTP connection (1): gssearch
gstask_1   | [2016-05-05 14:58:00,061: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[307ef8ab-e6f1-436a-ad78-cd9cd2935b95] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 14:58:00,061: INFO/Worker-5] Starting new HTTP connection (1): gssearch
gstask_1   | [2016-05-05 14:58:00,222: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[818cc652-e104-4426-850a-e771dddea37a] succeeded in 0.17372812s: None
gstask_1   | [2016-05-05 14:59:00,032: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 14:59:00,033: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 14:59:00,033: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[deb66e4c-42a3-4da5-a5d8-e4e47ddede90]
gstask_1   | 2016-05-05T14:59:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 14:59:00,034: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 14:59:00,036: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[4a47eeac-407b-43a3-b6e0-4814dbcec2ca]
gstask_1   | [2016-05-05 14:59:00,045: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[deb66e4c-42a3-4da5-a5d8-e4e47ddede90] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 14:59:00,138: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[4a47eeac-407b-43a3-b6e0-4814dbcec2ca] succeeded in 0.101007221s: None
gstask_1   | [2016-05-05 15:00:00,040: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:00:00,042: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[c34b2c94-956a-49da-a9eb-e46cdd130d20]
gstask_1   | 2016-05-05T15:00:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:00:00,043: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:00:00,051: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[c34b2c94-956a-49da-a9eb-e46cdd130d20] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:00:00,061: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:00:00,064: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[4a48e93d-d7be-4da7-9029-9ff740ed88fd]
gstask_1   | [2016-05-05 15:00:00,161: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[4a48e93d-d7be-4da7-9029-9ff740ed88fd] succeeded in 0.0961526950005s: None
gstask_1   | [2016-05-05 15:01:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:01:00,054: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:01:00,057: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[48a66ebb-a2b5-4598-ba28-8f51fa317bc5]
gstask_1   | [2016-05-05 15:01:00,058: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[0c9b7a15-9942-4b63-a279-30cc74969640]
gstask_1   | 2016-05-05T15:01:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:01:00,059: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:01:00,071: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[48a66ebb-a2b5-4598-ba28-8f51fa317bc5] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:01:00,171: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[0c9b7a15-9942-4b63-a279-30cc74969640] succeeded in 0.110544159s: None
gstask_1   | [2016-05-05 15:02:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:02:00,054: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:02:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[60c0ce58-fe7e-4794-a4b1-f983f67aa86b]
gstask_1   | [2016-05-05 15:02:00,057: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[8a9b5e85-4677-4dcc-9411-704363dde16d]
gstask_1   | 2016-05-05T15:02:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:02:00,057: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:02:00,069: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[60c0ce58-fe7e-4794-a4b1-f983f67aa86b] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:02:00,171: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[8a9b5e85-4677-4dcc-9411-704363dde16d] succeeded in 0.111637321s: None
gstask_1   | [2016-05-05 15:03:00,044: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:03:00,046: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:03:00,046: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[3bf9428a-4dd4-43c3-a582-03872d190cfa]
gstask_1   | 2016-05-05T15:03:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:03:00,047: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:03:00,048: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[460b3d0b-4795-4d4a-b4ae-d26bb77d1e16]
gstask_1   | [2016-05-05 15:03:00,055: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[3bf9428a-4dd4-43c3-a582-03872d190cfa] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:03:00,147: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[460b3d0b-4795-4d4a-b4ae-d26bb77d1e16] succeeded in 0.0981480309993s: None
gstask_1   | [2016-05-05 15:04:00,048: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:04:00,049: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[57e95880-0fb9-454d-b6c9-574e1f03b5d2]
gstask_1   | 2016-05-05T15:04:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:04:00,050: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:04:00,058: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[57e95880-0fb9-454d-b6c9-574e1f03b5d2] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:04:00,070: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:04:00,072: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[94d13d32-d9f4-4052-a458-5b2c8c8a4e70]
gstask_1   | [2016-05-05 15:04:00,157: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[94d13d32-d9f4-4052-a458-5b2c8c8a4e70] succeeded in 0.0843770430001s: None
gstask_1   | [2016-05-05 15:05:00,042: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:05:00,044: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:05:00,045: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[7fd64f5d-dcaf-409b-bc18-6ed4b740aa15]
gstask_1   | [2016-05-05 15:05:00,046: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[a6b6e390-ab21-4db6-9621-39d0c4264e8c]
gstask_1   | 2016-05-05T15:05:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:05:00,047: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:05:00,055: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[7fd64f5d-dcaf-409b-bc18-6ed4b740aa15] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:05:00,147: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[a6b6e390-ab21-4db6-9621-39d0c4264e8c] succeeded in 0.100138155s: None
gstask_1   | [2016-05-05 15:06:00,049: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:06:00,050: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:06:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[b2a8e0b9-e8a3-4aa4-8ce9-3c5d0868ce30]
gstask_1   | 2016-05-05T15:06:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:06:00,051: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:06:00,053: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[0e6785e2-5fbe-4056-a90a-fc2925cf4db7]
gstask_1   | [2016-05-05 15:06:00,061: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[b2a8e0b9-e8a3-4aa4-8ce9-3c5d0868ce30] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:06:00,158: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[0e6785e2-5fbe-4056-a90a-fc2925cf4db7] succeeded in 0.103206173s: None
gstask_1   | [2016-05-05 15:07:00,047: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:07:00,048: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:07:00,048: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[3ef3ca8e-24ef-46cd-a13d-ce05273fa497]
gstask_1   | 2016-05-05T15:07:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:07:00,049: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:07:00,050: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[13ddc95b-a447-43ef-859f-88c093861c87]
gstask_1   | [2016-05-05 15:07:00,059: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[3ef3ca8e-24ef-46cd-a13d-ce05273fa497] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:07:00,146: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[13ddc95b-a447-43ef-859f-88c093861c87] succeeded in 0.0940366239993s: None
gstask_1   | [2016-05-05 15:08:00,029: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:08:00,031: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[16a5e808-be57-4aca-8953-678a26055bc8]
gstask_1   | 2016-05-05T15:08:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:08:00,032: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:08:00,040: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[16a5e808-be57-4aca-8953-678a26055bc8] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:08:00,056: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:08:00,059: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[4a920d64-6316-47da-bae7-0fffb7c9245d]
gstask_1   | [2016-05-05 15:08:00,144: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[4a920d64-6316-47da-bae7-0fffb7c9245d] succeeded in 0.0848924290003s: None
gstask_1   | [2016-05-05 15:09:00,020: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:09:00,021: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:09:00,022: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[fb4c526b-d155-4ed6-810c-af2279a13db5]
gstask_1   | 2016-05-05T15:09:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:09:00,024: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[5a76355f-cb64-4088-85d6-67b500c7298a]
gstask_1   | [2016-05-05 15:09:00,022: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:09:00,034: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[fb4c526b-d155-4ed6-810c-af2279a13db5] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:09:00,121: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[5a76355f-cb64-4088-85d6-67b500c7298a] succeeded in 0.0972475080007s: None
gstask_1   | [2016-05-05 15:10:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:10:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:10:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[86447bec-1a61-4d53-aae8-f4e60838b3c8]
gstask_1   | [2016-05-05 15:10:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[b5feae93-c3d9-42d8-9cd5-bc29e3a6b440]
gstask_1   | 2016-05-05T15:10:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:10:00,055: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:10:00,064: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[86447bec-1a61-4d53-aae8-f4e60838b3c8] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:10:00,154: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[b5feae93-c3d9-42d8-9cd5-bc29e3a6b440] succeeded in 0.0991311509997s: None
gstask_1   | [2016-05-05 15:11:00,047: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:11:00,049: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:11:00,049: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[06e5f9ba-b40f-4b6c-b77b-16ef092b3650]
gstask_1   | 2016-05-05T15:11:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:11:00,050: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:11:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[2549d2e9-5948-4e3e-b0ab-6f6518adfa9f]
gstask_1   | [2016-05-05 15:11:00,059: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[06e5f9ba-b40f-4b6c-b77b-16ef092b3650] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:11:00,142: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[2549d2e9-5948-4e3e-b0ab-6f6518adfa9f] succeeded in 0.0906866340001s: None
gstask_1   | [2016-05-05 15:12:00,048: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:12:00,050: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[794c5844-4c64-434f-a2a8-5d50a661360c]
gstask_1   | 2016-05-05T15:12:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:12:00,051: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:12:00,059: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[794c5844-4c64-434f-a2a8-5d50a661360c] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:12:00,065: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:12:00,068: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[1802d6ab-645c-4844-ab03-868bc09be1a0]
gstask_1   | [2016-05-05 15:12:00,153: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[1802d6ab-645c-4844-ab03-868bc09be1a0] succeeded in 0.0843917299999s: None
gstask_1   | [2016-05-05 15:13:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:13:00,054: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:13:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[e468789d-3bde-4277-b373-46c558fb5669]
gstask_1   | [2016-05-05 15:13:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[c789edb4-444a-4688-a18b-105872c5e849]
gstask_1   | 2016-05-05T15:13:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:13:00,055: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:13:00,063: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[e468789d-3bde-4277-b373-46c558fb5669] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:13:00,153: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[c789edb4-444a-4688-a18b-105872c5e849] succeeded in 0.0971817380005s: None
gstask_1   | [2016-05-05 15:14:00,048: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | 2016-05-05T15:14:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:14:00,049: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:14:00,049: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[38c02190-ef25-4dd5-a396-40fc568e18a5]
gstask_1   | [2016-05-05 15:14:00,050: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:14:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[8d6883c5-5279-4a1f-b10b-cc635e46f8cb]
gstask_1   | [2016-05-05 15:14:00,059: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[38c02190-ef25-4dd5-a396-40fc568e18a5] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:14:00,152: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[8d6883c5-5279-4a1f-b10b-cc635e46f8cb] succeeded in 0.100446787s: None
gstask_1   | [2016-05-05 15:15:00,047: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:15:00,048: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:15:00,048: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[ef653599-a5a2-48c4-ac73-99ea5d64bdd1]
gstask_1   | 2016-05-05T15:15:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:15:00,049: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:15:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[1b817fb5-3c78-4001-9277-f199af5ab663]
gstask_1   | [2016-05-05 15:15:00,059: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[ef653599-a5a2-48c4-ac73-99ea5d64bdd1] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:15:00,152: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[1b817fb5-3c78-4001-9277-f199af5ab663] succeeded in 0.0993444249998s: None
^[[5~gstask_1   | [2016-05-05 15:16:00,024: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:16:00,029: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[2ab030df-d206-4e75-9ca2-7140ff7cb672]
gstask_1   | 2016-05-05T15:16:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:16:00,030: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:16:00,041: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:16:00,044: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[12b2a0d5-ed1b-4909-936e-c121b332432a]
gstask_1   | [2016-05-05 15:16:00,049: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[2ab030df-d206-4e75-9ca2-7140ff7cb672] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:16:00,235: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[12b2a0d5-ed1b-4909-936e-c121b332432a] succeeded in 0.188143271001s: None
^[[5~^[[5~^[[5~^[[5~gstask_1   | [2016-05-05 15:17:00,049: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:17:00,051: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:17:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[b0524ce7-22d0-467c-884b-8a7a1eadf14b]
gstask_1   | 2016-05-05T15:17:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:17:00,052: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:17:00,053: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[6be2bca9-4da9-46ff-a169-5899fc74d950]
gstask_1   | [2016-05-05 15:17:00,064: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[b0524ce7-22d0-467c-884b-8a7a1eadf14b] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:17:00,150: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[6be2bca9-4da9-46ff-a169-5899fc74d950] succeeded in 0.0964250850002s: None
gstask_1   | [2016-05-05 15:18:00,049: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:18:00,050: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:18:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[cb96d998-1c3f-4abb-be55-f150addafe44]
gstask_1   | 2016-05-05T15:18:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:18:00,052: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:18:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[eb20f7eb-e41f-4f59-9b16-cc185f6ad50e]
gstask_1   | [2016-05-05 15:18:00,061: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[cb96d998-1c3f-4abb-be55-f150addafe44] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:18:00,161: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[eb20f7eb-e41f-4f59-9b16-cc185f6ad50e] succeeded in 0.105405583s: None
gstask_1   | [2016-05-05 15:19:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:19:00,057: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[f3027a5b-58ae-42ee-8885-fa46cbe9d510]
gstask_1   | 2016-05-05T15:19:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:19:00,058: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:19:00,069: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[f3027a5b-58ae-42ee-8885-fa46cbe9d510] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:19:00,072: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:19:00,075: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[563cdf15-34a3-4131-877f-7434a72f3080]
gstask_1   | [2016-05-05 15:19:00,183: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[563cdf15-34a3-4131-877f-7434a72f3080] succeeded in 0.105932507s: None
gstask_1   | [2016-05-05 15:20:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:20:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:20:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[7f454320-9a2d-4fd2-b74b-9cd1b292b798]
gstask_1   | 2016-05-05T15:20:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:20:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[a69bf66a-0dfd-4bf5-84e4-f1648dc9b374]
gstask_1   | [2016-05-05 15:20:00,057: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:20:00,067: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[7f454320-9a2d-4fd2-b74b-9cd1b292b798] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:20:00,194: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[a69bf66a-0dfd-4bf5-84e4-f1648dc9b374] succeeded in 0.136356907999s: None
gstask_1   | [2016-05-05 15:21:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:21:00,054: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:21:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[b1311a41-e42e-47a1-8f8d-8cfcb1a4357f]
gstask_1   | 2016-05-05T15:21:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:21:00,055: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:21:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[98178cce-31d3-462d-86e4-7b63ddcd7ce1]
gstask_1   | [2016-05-05 15:21:00,065: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[b1311a41-e42e-47a1-8f8d-8cfcb1a4357f] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:21:00,182: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[98178cce-31d3-462d-86e4-7b63ddcd7ce1] succeeded in 0.125149845s: None
gstask_1   | [2016-05-05 15:22:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:22:00,054: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:22:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[49922cb7-f196-4cca-a157-391572063fc9]
gstask_1   | 2016-05-05T15:22:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:22:00,055: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:22:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[f46a62c2-d2a8-459f-8c3a-6c96b3dae6f6]
gstask_1   | [2016-05-05 15:22:00,066: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[49922cb7-f196-4cca-a157-391572063fc9] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:22:00,203: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[f46a62c2-d2a8-459f-8c3a-6c96b3dae6f6] succeeded in 0.146382554s: None
gstask_1   | [2016-05-05 15:23:00,045: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:23:00,047: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[d27a5616-5bca-4730-b19e-41eab1342e59]
gstask_1   | 2016-05-05T15:23:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:23:00,048: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:23:00,055: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[d27a5616-5bca-4730-b19e-41eab1342e59] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:23:00,070: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:23:00,072: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[6d0d12b0-646c-45e1-82b9-3c999961e911]
gstask_1   | [2016-05-05 15:23:00,169: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[6d0d12b0-646c-45e1-82b9-3c999961e911] succeeded in 0.0966967409995s: None
gstask_1   | [2016-05-05 15:24:00,053: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:24:00,058: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:24:00,059: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[9a44a892-30c6-4af7-aede-866659ea8c56]
gstask_1   | 2016-05-05T15:24:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:24:00,061: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:24:00,063: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[ae4d5f00-0aa3-403f-a81e-6dd1399b860c]
gstask_1   | [2016-05-05 15:24:00,078: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[9a44a892-30c6-4af7-aede-866659ea8c56] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:24:00,284: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[ae4d5f00-0aa3-403f-a81e-6dd1399b860c] succeeded in 0.218323516s: None
gstask_1   | [2016-05-05 15:25:00,051: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:25:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:25:00,053: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[6420f818-85a9-49a1-bfe0-01d14faad581]
gstask_1   | 2016-05-05T15:25:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:25:00,054: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:25:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[d03d72d8-462e-43be-939f-53edc834bc6a]
gstask_1   | [2016-05-05 15:25:00,063: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[6420f818-85a9-49a1-bfe0-01d14faad581] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:25:00,158: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[d03d72d8-462e-43be-939f-53edc834bc6a] succeeded in 0.102296853s: None
gstask_1   | [2016-05-05 15:26:00,050: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:26:00,052: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:26:00,052: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[79250fc1-b28a-4d59-9418-69a518d54ad9]
gstask_1   | [2016-05-05 15:26:00,053: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[36be8a77-e22d-4f65-9e09-9d81c908fe4d]
gstask_1   | 2016-05-05T15:26:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:26:00,053: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:26:00,063: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[79250fc1-b28a-4d59-9418-69a518d54ad9] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:26:00,156: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[36be8a77-e22d-4f65-9e09-9d81c908fe4d] succeeded in 0.102984047s: None
gstask_1   | [2016-05-05 15:27:00,036: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:27:00,039: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[8ef29d42-c01f-4d45-8b8a-47ef060fafa0]
gstask_1   | 2016-05-05T15:27:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:27:00,039: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:27:00,048: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[8ef29d42-c01f-4d45-8b8a-47ef060fafa0] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:27:00,056: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:27:00,058: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[18657a86-4f4b-4947-820e-2c8e2c2c2385]
gstask_1   | [2016-05-05 15:27:00,156: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[18657a86-4f4b-4947-820e-2c8e2c2c2385] succeeded in 0.0963752750004s: None
gstask_1   | [2016-05-05 15:28:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:28:00,055: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | 2016-05-05T15:28:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:28:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[9818f52c-0954-42c6-a740-7f9527d495a1]
gstask_1   | [2016-05-05 15:28:00,056: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:28:00,058: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[792a9391-bba3-4785-a0b9-9ad8dc613170]
gstask_1   | [2016-05-05 15:28:00,069: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[9818f52c-0954-42c6-a740-7f9527d495a1] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:28:00,179: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[792a9391-bba3-4785-a0b9-9ad8dc613170] succeeded in 0.119129257s: None
gstask_1   | [2016-05-05 15:29:00,053: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:29:00,055: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:29:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[894b0e9f-8eb8-4722-9792-422c3a4ba84e]
gstask_1   | [2016-05-05 15:29:00,057: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[cdf86dce-b900-4efa-a65c-1dac42b9dbbe]
gstask_1   | 2016-05-05T15:29:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:29:00,058: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:29:00,070: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[894b0e9f-8eb8-4722-9792-422c3a4ba84e] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:29:00,166: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[cdf86dce-b900-4efa-a65c-1dac42b9dbbe] succeeded in 0.107908971s: None
gstask_1   | [2016-05-05 15:30:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:30:00,056: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:30:00,064: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[7af6a5d7-0cd1-4daa-b4f2-5333b8b32aef]
gstask_1   | [2016-05-05 15:30:00,067: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[6dc219db-26c4-4927-80b1-263598bb75a8]
gstask_1   | 2016-05-05T15:30:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:30:00,068: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:30:00,092: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[7af6a5d7-0cd1-4daa-b4f2-5333b8b32aef] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:30:00,234: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[6dc219db-26c4-4927-80b1-263598bb75a8] succeeded in 0.164710174s: None
gstask_1   | [2016-05-05 15:31:00,080: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:31:00,081: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:31:00,082: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[a8930a0c-5793-435d-93f0-b3303403f0ec]
gstask_1   | 2016-05-05T15:31:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:31:00,083: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:31:00,084: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[f4399fcd-b606-4332-a417-73dcfc8f964f]
gstask_1   | [2016-05-05 15:31:00,092: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[a8930a0c-5793-435d-93f0-b3303403f0ec] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:31:00,176: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[f4399fcd-b606-4332-a417-73dcfc8f964f] succeeded in 0.0908578699991s: None
gstask_1   | [2016-05-05 15:32:00,049: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:32:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:32:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[31f4486f-fb20-4149-93dd-bfd57bc12d54]
gstask_1   | [2016-05-05 15:32:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[cbe989fc-aaa8-4dc9-b1eb-1a868db6cbad]
gstask_1   | 2016-05-05T15:32:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:32:00,056: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:32:00,075: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[31f4486f-fb20-4149-93dd-bfd57bc12d54] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:32:00,211: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[cbe989fc-aaa8-4dc9-b1eb-1a868db6cbad] succeeded in 0.154153474s: None
gstask_1   | [2016-05-05 15:33:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:33:00,055: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:33:00,057: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[6ce35438-e7ab-4c05-9555-ab63e12784bf]
gstask_1   | [2016-05-05 15:33:00,059: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[45a4a07f-b11d-4d1f-b22a-6d70f25c9bcc]
gstask_1   | 2016-05-05T15:33:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:33:00,061: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:33:00,077: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[6ce35438-e7ab-4c05-9555-ab63e12784bf] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:33:00,265: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[45a4a07f-b11d-4d1f-b22a-6d70f25c9bcc] succeeded in 0.201716776s: None
gstask_1   | [2016-05-05 15:34:00,051: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:34:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[74669cd4-9680-404e-b135-08a4e7d5887e]
gstask_1   | 2016-05-05T15:34:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:34:00,055: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:34:00,063: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:34:00,066: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[41a6062c-3ecf-44a0-9591-b8b454396c7c]
gstask_1   | [2016-05-05 15:34:00,067: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[74669cd4-9680-404e-b135-08a4e7d5887e] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:34:00,174: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[41a6062c-3ecf-44a0-9591-b8b454396c7c] succeeded in 0.106236866s: None
gstask_1   | [2016-05-05 15:35:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:35:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:35:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[296fdf2c-d9c1-41a8-8cda-13397c358d4f]
gstask_1   | 2016-05-05T15:35:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:35:00,056: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:35:00,058: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[0c9aba0c-3f31-49e3-b61f-56a84556b25f]
gstask_1   | [2016-05-05 15:35:00,077: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[296fdf2c-d9c1-41a8-8cda-13397c358d4f] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:35:00,174: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[0c9aba0c-3f31-49e3-b61f-56a84556b25f] succeeded in 0.114914549s: None
gstask_1   | [2016-05-05 15:36:00,048: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:36:00,049: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:36:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[703035c3-0dfc-42f2-9d8c-a568f8c0160a]
gstask_1   | [2016-05-05 15:36:00,052: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:36:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[3a98c808-cf0c-49a2-9858-4f249d02cd4b]
gstask_1   | 2016-05-05T15:36:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:36:00,062: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[703035c3-0dfc-42f2-9d8c-a568f8c0160a] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:36:00,197: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[3a98c808-cf0c-49a2-9858-4f249d02cd4b] succeeded in 0.141886101s: None
gstask_1   | [2016-05-05 15:37:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:37:00,056: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:37:00,057: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[b13fe759-14d4-4585-a095-921bd236469a]
gstask_1   | 2016-05-05T15:37:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:37:00,059: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:37:00,063: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[6a833057-4536-4be0-8993-2e3370dab017]
gstask_1   | [2016-05-05 15:37:00,072: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[b13fe759-14d4-4585-a095-921bd236469a] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:37:00,504: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[6a833057-4536-4be0-8993-2e3370dab017] succeeded in 0.440637336999s: None
gstask_1   | [2016-05-05 15:38:00,053: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:38:00,057: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[53b2e42a-d7ba-4d28-905f-18fc14d541a3]
gstask_1   | 2016-05-05T15:38:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:38:00,060: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:38:00,071: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:38:00,074: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[53b2e42a-d7ba-4d28-905f-18fc14d541a3] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:38:00,075: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[26db8f8a-e4cc-4800-8a18-8863c7a83fb5]
gstask_1   | [2016-05-05 15:38:00,195: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[26db8f8a-e4cc-4800-8a18-8863c7a83fb5] succeeded in 0.119650612s: None
gstask_1   | [2016-05-05 15:39:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:39:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:39:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[97c4e587-2b87-4b73-9918-fe02f3bbf2de]
gstask_1   | [2016-05-05 15:39:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[ce82c42e-d726-42a1-81f5-a61a871521e7]
gstask_1   | 2016-05-05T15:39:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:39:00,056: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:39:00,071: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[97c4e587-2b87-4b73-9918-fe02f3bbf2de] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:39:00,171: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[ce82c42e-d726-42a1-81f5-a61a871521e7] succeeded in 0.113992004s: None
gstask_1   | [2016-05-05 15:40:00,048: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:40:00,049: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:40:00,050: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[98c83b55-c9fb-4544-9393-cc034c1ef0e9]
gstask_1   | [2016-05-05 15:40:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[7ba72027-1473-45c9-b7df-5c54de7aa424]
gstask_1   | 2016-05-05T15:40:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:40:00,051: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:40:00,062: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[98c83b55-c9fb-4544-9393-cc034c1ef0e9] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:40:00,160: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[7ba72027-1473-45c9-b7df-5c54de7aa424] succeeded in 0.109154606s: None
gstask_1   | [2016-05-05 15:41:00,047: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:41:00,050: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:41:00,050: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[25e23928-2be9-45f2-9e84-359039968564]
gstask_1   | 2016-05-05T15:41:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:41:00,052: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:41:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[53b48d5c-6a3f-4cc5-8e7b-605149a8bcbc]
gstask_1   | [2016-05-05 15:41:00,063: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[25e23928-2be9-45f2-9e84-359039968564] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:41:00,183: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[53b48d5c-6a3f-4cc5-8e7b-605149a8bcbc] succeeded in 0.127709578s: None
gstask_1   | [2016-05-05 15:42:00,051: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:42:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[f98a7a0f-6d21-4ef3-9bbd-4a19016a1155]
gstask_1   | 2016-05-05T15:42:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:42:00,055: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:42:00,070: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:42:00,076: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[c1f4ad69-2a5a-48b1-8d05-eba2d19977a9]
gstask_1   | [2016-05-05 15:42:00,078: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[f98a7a0f-6d21-4ef3-9bbd-4a19016a1155] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:42:00,203: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[c1f4ad69-2a5a-48b1-8d05-eba2d19977a9] succeeded in 0.123816097001s: None
gstask_1   | [2016-05-05 15:43:00,053: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:43:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[b21b5b24-efd2-4609-ba93-e654ecf9479f]
gstask_1   | [2016-05-05 15:43:00,055: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:43:00,056: INFO/Worker-3] Starting service status check
gstask_1   | 2016-05-05T15:43:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:43:00,060: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[6e9c7d3e-eda3-443f-9518-239885062163]
gstask_1   | [2016-05-05 15:43:00,067: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[b21b5b24-efd2-4609-ba93-e654ecf9479f] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:43:00,168: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[6e9c7d3e-eda3-443f-9518-239885062163] succeeded in 0.10793862s: None
gstask_1   | [2016-05-05 15:44:00,049: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:44:00,051: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:44:00,052: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[733bca0b-636a-4785-8b35-a02bfe0153f3]
gstask_1   | 2016-05-05T15:44:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:44:00,054: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:44:00,059: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[73e15daa-5bde-4ba8-837b-c99a8e6ec7d7]
gstask_1   | [2016-05-05 15:44:00,107: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[733bca0b-636a-4785-8b35-a02bfe0153f3] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:44:00,214: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[73e15daa-5bde-4ba8-837b-c99a8e6ec7d7] succeeded in 0.15313921s: None
gstask_1   | [2016-05-05 15:45:00,051: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:45:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | 2016-05-05T15:45:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:45:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[a976ff19-074b-4d54-be1b-48d34cd6c931]
gstask_1   | [2016-05-05 15:45:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[7a5faa70-5a42-4a0b-aee9-759fb45e69da]
gstask_1   | [2016-05-05 15:45:00,057: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:45:00,068: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[a976ff19-074b-4d54-be1b-48d34cd6c931] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:45:00,189: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[7a5faa70-5a42-4a0b-aee9-759fb45e69da] succeeded in 0.132207549001s: None
gstask_1   | [2016-05-05 15:46:00,048: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | 2016-05-05T15:46:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:46:00,052: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[14be6d47-53da-4477-b04e-7dfc1e687a51]
gstask_1   | [2016-05-05 15:46:00,053: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:46:00,064: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[14be6d47-53da-4477-b04e-7dfc1e687a51] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:46:00,066: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:46:00,068: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[5ace6686-5479-4f86-9802-e4083a580cc7]
gstask_1   | [2016-05-05 15:46:00,178: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[5ace6686-5479-4f86-9802-e4083a580cc7] succeeded in 0.108501563s: None
gstask_1   | [2016-05-05 15:47:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:47:00,054: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:47:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[ffeac1bd-daba-481e-9d3a-fcbdd0bae785]
gstask_1   | 2016-05-05T15:47:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:47:00,055: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:47:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[89195b44-c89a-4ff1-bdd1-91332f384bb6]
gstask_1   | [2016-05-05 15:47:00,065: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[ffeac1bd-daba-481e-9d3a-fcbdd0bae785] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:47:00,177: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[89195b44-c89a-4ff1-bdd1-91332f384bb6] succeeded in 0.121086540999s: None
gstask_1   | [2016-05-05 15:48:00,049: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:48:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:48:00,053: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[e5e36938-15b8-430d-b7a4-5327e814d41a]
gstask_1   | 2016-05-05T15:48:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:48:00,054: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:48:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[cce4364b-45b0-4781-9af4-31e5cd885b50]
gstask_1   | [2016-05-05 15:48:00,064: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[e5e36938-15b8-430d-b7a4-5327e814d41a] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:48:00,199: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[cce4364b-45b0-4781-9af4-31e5cd885b50] succeeded in 0.142924754s: None
gstask_1   | [2016-05-05 15:49:00,047: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:49:00,049: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:49:00,049: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[158070ec-a675-4755-81b0-f7b75b8ba9c3]
gstask_1   | 2016-05-05T15:49:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:49:00,050: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:49:00,051: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[78cfcf0a-94e1-48d3-8ddb-f8ee9cf9472c]
gstask_1   | [2016-05-05 15:49:00,060: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[158070ec-a675-4755-81b0-f7b75b8ba9c3] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:49:00,187: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[78cfcf0a-94e1-48d3-8ddb-f8ee9cf9472c] succeeded in 0.135285567s: None
gstask_1   | [2016-05-05 15:50:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:50:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[b0e26cc6-77cb-4328-8c78-d8b56557c3cc]
gstask_1   | [2016-05-05 15:50:00,055: INFO/Worker-2] Starting service status check
gstask_1   | 2016-05-05T15:50:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:50:00,067: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[b0e26cc6-77cb-4328-8c78-d8b56557c3cc] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:50:00,075: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:50:00,078: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[337c98ab-6fd5-462e-b0ad-6fe80754c9fc]
gstask_1   | [2016-05-05 15:50:00,187: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[337c98ab-6fd5-462e-b0ad-6fe80754c9fc] succeeded in 0.108759659s: None
gstask_1   | [2016-05-05 15:51:00,053: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:51:00,054: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:51:00,054: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[7050a36c-11e0-4318-9647-e91ddce365bc]
gstask_1   | [2016-05-05 15:51:00,055: INFO/Worker-3] Starting service status check
gstask_1   | 2016-05-05T15:51:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:51:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[580ef571-ea00-4522-99a9-5dd3f187de3a]
gstask_1   | [2016-05-05 15:51:00,067: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[7050a36c-11e0-4318-9647-e91ddce365bc] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:51:00,164: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[580ef571-ea00-4522-99a9-5dd3f187de3a] succeeded in 0.106665661s: None
gstask_1   | [2016-05-05 15:52:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:52:00,054: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:52:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[06c81254-ed3a-4eb8-ace9-45a6987c8045]
gstask_1   | [2016-05-05 15:52:00,057: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[c90cb213-d142-463a-b465-16fc6c31cd26]
gstask_1   | 2016-05-05T15:52:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:52:00,058: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:52:00,069: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[06c81254-ed3a-4eb8-ace9-45a6987c8045] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:52:00,165: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[c90cb213-d142-463a-b465-16fc6c31cd26] succeeded in 0.10638979s: None
gstask_1   | [2016-05-05 15:53:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:53:00,055: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:53:00,061: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[52864137-56ee-405b-90b6-35b08a32d67c]
gstask_1   | [2016-05-05 15:53:00,065: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[a697c4c1-2217-4356-aeb8-f08701345ef9]
gstask_1   | 2016-05-05T15:53:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:53:00,066: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:53:00,092: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[52864137-56ee-405b-90b6-35b08a32d67c] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:53:00,233: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[a697c4c1-2217-4356-aeb8-f08701345ef9] succeeded in 0.166038904s: None
gstask_1   | [2016-05-05 15:54:00,051: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:54:00,053: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[63b42153-f02a-4495-aa74-07966a5a8929]
gstask_1   | 2016-05-05T15:54:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:54:00,054: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:54:00,061: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:54:00,064: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[117f1be3-81e6-48ef-a042-3d6aa0636fa8]
gstask_1   | [2016-05-05 15:54:00,065: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[63b42153-f02a-4495-aa74-07966a5a8929] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:54:00,162: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[117f1be3-81e6-48ef-a042-3d6aa0636fa8] succeeded in 0.096832278s: None
gstask_1   | [2016-05-05 15:55:00,052: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:55:00,053: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:55:00,053: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[a8847837-03b1-4cd7-9221-3355f81f2500]
gstask_1   | 2016-05-05T15:55:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:55:00,054: INFO/Worker-3] Starting service status check
gstask_1   | [2016-05-05 15:55:00,055: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[453b38f3-d756-46cd-b9b8-259489984363]
gstask_1   | [2016-05-05 15:55:00,067: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[a8847837-03b1-4cd7-9221-3355f81f2500] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:55:00,174: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[453b38f3-d756-46cd-b9b8-259489984363] succeeded in 0.118153369s: None
gstask_1   | [2016-05-05 15:56:00,048: INFO/Beat] Scheduler: Sending due task service_status_check (goldstone.core.tasks.service_status_check)
gstask_1   | [2016-05-05 15:56:00,050: INFO/Beat] Scheduler: Sending due task process_alerts (goldstone.core.tasks.process_alerts)
gstask_1   | [2016-05-05 15:56:00,053: INFO/MainProcess] Received task: goldstone.core.tasks.service_status_check[d94094a1-c276-49ce-9375-26d3394895ec]
gstask_1   | [2016-05-05 15:56:00,056: INFO/MainProcess] Received task: goldstone.core.tasks.process_alerts[a6541f10-edd8-4f55-97ba-d938a46e0166]
gstask_1   | 2016-05-05T15:56:00+0000 INFO     goldstone.core.tasks Starting service status check
gstask_1   | [2016-05-05 15:56:00,057: INFO/Worker-2] Starting service status check
gstask_1   | [2016-05-05 15:56:00,085: ERROR/MainProcess] Task goldstone.core.tasks.service_status_check[d94094a1-c276-49ce-9375-26d3394895ec] raised unexpected: KeyError('aggregations',)
gstask_1   | Traceback (most recent call last):
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gstask_1   |     R = retval = fun(*args, **kwargs)
gstask_1   |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in protected_call
gstask_1   |     return self.run(*args, **kwargs)
gstask_1   |   File "/home/app/goldstone/core/tasks.py", line 190, in service_status_check
gstask_1   |     found_services_by_host = result['aggregations']['per_host']['buckets']
gstask_1   | KeyError: 'aggregations'
gstask_1   | [2016-05-05 15:56:00,230: INFO/MainProcess] Task goldstone.core.tasks.process_alerts[a6541f10-edd8-4f55-97ba-d938a46e0166] succeeded in 0.170811331s: None

we should do some minimal checking on the ES payload and handle this condition better than spewing key error exceptions to the logs.

seems there are several of these conditions.  deferring fix until we can look at solving it for all of them.  Possibly can do that through the DRF global exception processing mechanism. Moving to backlog.