Add a flag to exclude tainted nodes

On a cluster that has many tainted nodes it's not straightforward to understand whether there is sufficient capacity to drain one: nodes with taints can make it seem like there is more capacity than there really would be.
This patch solves for the simplest use case by allowing all tainted nodes to be excluded with a flag.

Thanks @barrykp! Sorry for the delay on getting to this!


Add initial structure and ballerina based proxy

Purpose

Introduce the initial structure of the repository and add the ballerina based proxy.

Goals

Introduce a base structure to work on top of.

Approach

Added Travis, Codecov, Makefile and other structure related content. Also added the ballerina based proxy along with its Dockerfile and Portal's Dockerfile and Docker compose config.

User stories

N/A

Release note

N/A

Documentation

N/A

Training

N/A

Certification

N/A.

Marketing

N/A

Automation tests

Unit tests

N/A

Integration tests

N/A

Security checks

Followed secure coding standards in http://wso2.com/technical-reports/wso2-secure-engineering-guidelines? yes
Ran FindSecurityBugs plugin and verified report? yes
Confirmed that this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets? yes

Samples

N/A

Related PRs

N/A

Migrations (if applicable)

N/A

Test environment

Docker version 18.06.1-ce, build e68fc7a
GNU Make 4.1

Learning

N/A

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@1cd213b). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master      #2   +/-   ##
========================================
  Coverage          ?   35.2%
========================================
  Files             ?      13
  Lines             ?     125
  Branches          ?      17
========================================
  Hits              ?      44
  Misses            ?      71
  Partials          ?      10

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1cd213b...95026e7. Read the comment docs.


Horishontal scrollbar in index page

in index page there is a horishontal scroll bar which is not looking good and its irritating too

@TeeWrath i will solve assign it to me

@TeeWrath i will be grateful if you assign me with this issue

@TeeWrath can you it to me . I will surely fix the issue of horizontal scroll bar .


2024 aux discrete

This concludes the added auxiliary sensors functionality (#57 ) and a bunch of user warnings based on discrete reft, salt, and oxy data files (#58 ).
Re-merged the test fixes from the flagging update.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 23.46939% with 75 lines in your changes missing coverage. Please review.

Files with missing lines
Patch %
Lines

ctdcal/oxy_fitting.py
3.33%
29 Missing :warning:

ctdcal/process_bottle.py
3.44%
28 Missing :warning:

ctdcal/odf_io.py
37.93%
11 Missing and 7 partials :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Flag
Coverage Δ

pytest
29.95% <23.46%> (-3.93%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines
Coverage Δ

ctdcal/config.py
100.00% <ø> (ø)

ctdcal/convert.py
10.21% <ø> (ø)

ctdcal/equations_sbe.py
97.29% <100.00%> (+0.26%)
:arrow_up:

ctdcal/odf_io.py
72.72% <37.93%> (-8.59%)
:arrow_down:

ctdcal/process_bottle.py
22.13% <3.44%> (-2.32%)
:arrow_down:

ctdcal/oxy_fitting.py
16.38% <3.33%> (-1.21%)
:arrow_down:

... and 2 files with indirect coverage changes


Using OMP with Rcpp in Rstudio in windows

Dear all,
I wonder if it is possible to use OMP with Rcpp in windows using Rstudio. I wrote a simple code, but when I compile it, it gives me a warring message telling me that all pragmas lines are ignored.
Any idea about how to fix it??
Regards

@choushishi
@ahmedahmedov

Try installing Rtools.

How is this issue related to xgboost?
But make sure you have OpenMP flags enabled in Makevars.win, like in https://github.com/dmlc/xgboost/blob/master/R-package/src/Makevars.win


fixes a bug in which complex options are not passed through chain

What kind of change does this PR introduce?
bugfix
Did you add tests for your changes?
yes
If relevant, did you update the documentation?
N/A
Summary
pitching loaders get the remainingRequest passed. As this is a string complex options doesn't get passed. Instead the ident is passed. This PR fixes the conversion from ident back to complex options.

Does this PR introduce a breaking change?
no, it's a bugfix

#2901


Integrate how to get WMTS endpoints

From https://github.com/hotosm/OpenAerialMap/issues/59, we will be enabling WMTS endpoints for all imagery processed through the Uploader. Right now we only expose the TMS endpoint through the dropdown selector and enable users to either open in an editor or copy to clipboard.
I think we want to keep this interaction but add on some way to get the WMTS url and copy to clipboard.

Ideas for how we can handle this?

Although technically simple, this poses some challenges UX/UI wise.
This would probably be another field with a url, right?

Yeah, I guess the options are:
Split the button:

Copy to clipboard: TMS | WMTS

or
Add another field:

Copy TMS to clipboard
Copy WMTS to clipboard

Other ideas or options?

I assume the url is different right?
It's not good ux if we have a field with some text, and the when clicking copy, it copies something different (event if wmts is written). Those options are contextual of that field. Does this make sense?

If I follow you correctly, the issue you're saying is that the dropdown is related the TMS which you can see partially in the element. If we add another item on the dropdown that even if we label it, it may not be clear to the user that they are copying something different?
The urls are different but only slightly. These are the differences:
http://tiles.staging.openaerialmap.org/5885c3bb39240e00117b78c6/0/31d10d04-c626-4867-ba51-39deaf61337c/{z}/{x}/{y}.png
http://tiles.staging.openaerialmap.org/5885c3bb39240e00117b78c6/0/31d10d04-c626-4867-ba51-39deaf61337c/wmts

Exactly!
I'd say we could add another field, but this would clutter the interface, or maybe remove the field and add a dropdown button for tms (with the existing options) and another one for wmts (only with copy).

I feel like adding a second dropdown adds a lot of complexity when the two functions are very similar.
What if we did this:

change the field text to be the main part of the URL which is the tilejson url: http://tiles.staging.openaerialmap.org/5885c3bb39240e00117b78c6/0/31d10d04-c626-4867-ba51-39deaf61337c
Then have four dropdown actions:

Open in iD
Open in JOSM
Copy TMS
Copy WMTS

Yes, I like that, mainly because it triples as the TileJSON URL.

So maybe "Copy TileJSON" as well? And add "URL" to each "Copy ..." entry?

Made an attempt:

Works for me, in part because the full URL is usually not visible anyway.

After discussing this a bit with @ricardomestre adding the option under the dropdown with a small separator helps mitigate the issue. It's not a perfect solution but will do. Implemented in #196


Prevents indent on blank lines of partials.

This pull request is to prevent indent in partials that have blank lines. It is needed when handlebars is used as a code generator and the spacing of the output is important or linted.
Here is the result of the test in the pull request before the fix:

The blank line in Yehuda's function has indent. This is not desirable. After the fix, this line is no longer indented.

[x] Please don't start pull requests for security issues. Instead, file a report at https://www.npmjs.com/advisories/report?package=handlebars
[x] Maintain the existing code style
[x] Are focused on a single change (i.e. avoid large refactoring or style adjustments in untouched code if not the primary goal of the pull request)
[x] Have good commit messages
[x] Have tests
[x] Have the typings (lib/handlebars.d.ts) updated on every API change. If you need help, updating those, please mention that in the PR description.
[x] Don't significantly decrease the current code coverage (see coverage/lcov-report/index.html)
[x] Currently, the 4.x-branch contains the latest version. Please target that branch in the PR.

Sounds like a breaking change to me, and a pretty special use-case. It should at least be a feature switch.
Even then... Couldn't you just add a post-processing step like
const withoutEmptySpaces = string.replace(/^\s+$/gm,"")

or use prettier to format the result? Seems easier to me than changing Handlebars.

Thanks for taking a look.
Here is some output which is bit more illustrative of our use case.

We really see this as a bug in the indentation algorithm. We feel it's not clean code to add indent on the blank line of the function above. Anybody generating code that will be linted would benefit from this fix.
What use case would want indent for blank lines of a partial?
We could post-process, as you recommend, but we use this extensively at https://www.uvmgen.com and since we are reactively generating code on the front end, performance is important to us. This does not add any performance overhead to handlebars since it already has a condition checking for blank lines as it adds indent.
I'm not understanding this being a breaking change.
Adding indent originally (https://github.com/wycats/handlebars.js/issues/858) needed the preventDefault feature switch because indentation was a new feature. If you consider this a bug fix as we do, a feature switch may not be needed.

I have become very cautious with changes and very cautious with the assumption that "nobody would use Handlebars this way". The lack of a formal language specification makes it difficult to say what is a bug and what isn't. The large number of users makes it probable that one of them relies on exactly that behavior that you want to change, even if it seems insignificant to you and me.
You are right that when using Handlebars to create JavaScript, Java or alike, it seems natural to skip indent on blank lines. In Python, the majority also seems to prefer blank lines without indent. But there are some that say blank lines should be indented (https://stackoverflow.com/a/25002433).
Without any target file type in mind, treating blank lines and non-blank lines the same seems natural to me, so I don't consider this a bug.
I cannot say for sure that your change does or doesn't break any build, except by publishing a new version with the change. Then, if just one user complains, I will have to revert the change and publish a new version. The user may have had an hour of debugging until she finds out that a change in Handlebars is responsible for her test failures. And I have spent at least an hour publishing two patch versions (publishing is not completely automated yet).
But since this affects that language, I'd like to get @wycats opinion, too.

Oh, it just came to my mind that if this is not a bug, than it is a new feature that changes the language. I promised @wycats not to allow any new features, until there is a formal language specification.
Another reason to delegate this decision to @wycats

@nknapp thanks for being cautious!
I tend to think that this is on the border between a feature and a bug, but you're right that it's very likely to end up being a breaking change.
I wouldn't be opposed to collecting use-cases for a new revision of the whitespace stripping algorithm (indentation sensitivity comes to mind as a justification for a revision), and in that context, I think this change would make sense.


refactor: use cobra stdout for check subcommand

uses cobra's stdout rather than the global stdout so we can test things more easily

marking as a refactor because this doesn't affect external anything


Add some initial styles for vaadin-month-calendar

Closing this for now. Will add another PR later with more comprehensive styling.


Placeholder not returning to show in IOS

I am having small UI/UX issue with the place holders for the braintree hosted fields implementation. This may not be a braintree issue perse since I created this but this could cause other issues as well so I wanted to mention it.

SDK version: 3.30.0
Environment: IOS 11.2.2
Browser and OS: Safai & IOS 11.2.2

When you have placeholders on you form fields it works as expected but when you do not enter anything and you click another fields the placeholder does not come back.
I do not have labels on my fields and just placeholders so when it is blank users would not know what the fields is for.
I wanted the fields to be blank when you focus on the field and have the placeholder disappear because users may think they need to remove the placeholder in the field becuase it looks like text.
However when removing the focus it still thinks it is focused in the iFrame.
Here is a snippet of the code
client : clientInstance,
						styles :
						{
							'select, input' :
							{
								'font-size' : '15px',
								'font-family' : '"Open Sans", sans-serif',
								'color' : '#727070',
								'padding' : '0 0 0 18px'
							},
							'input:focus::placeholder' :
							{
								'opacity' : '0'
							}
						},
						fields :
						{
							number :
							{
								selector : '#credit-card',
								placeholder : 'Credit Card Number'
							},
							cvv :
							{
								selector : '#cvv',
								placeholder : 'CVV'
							},
							expirationMonth :
							{
								selector : '#ex-month',
								placeholder : 'Month',
								select :
								{
									options :
									[ '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12' ]
								}
							},
							expirationYear :
							{
								selector : '#ex-year',
								placeholder : 'Year',
								select : true
							},
							postalCode :
							{
								selector : '#zipcode',
								placeholder : 'Postal Code'
							}
						}

This is, unfortunately, a bug in iOS Safari where the focus state isn't interpreted correctly when inside an iframe. We've tried to build workarounds for this, but it's impossible to cover it completely.
One thing you can do is report it to Apple so that, eventually, they do fix the bug.


Verify the spec formally with a reference implementation

To validate things like #5, I think it would be really good to actually implement the spec exactly, and then use that against a series of comprehensive test cases. https://github.com/tc39/proposal-promise-finally/blob/master/polyfill.js does not follow the spec language at all so it is not helpful for this.
The easiest way to do this is probably to start with https://github.com/domenic/cancelable-promise/blob/bc4d207e7bcd3e7feb4bf6db3edc42d23c683051/lib/promise.js

In general i try to follow spec language in polyfill implementations, but since the polyfill here is essentially hijacking existing methods, and most of the spec changes aren't observable in JS, i'm not sure how well that will work. I'll definitely look into it though, thanks!

Right, the idea is to not have a polyfill that extends existing promise implementations, but instead a reference implementation of the entire promise spec + this spec, to verify that the spec is correct. That would make reviews a lot easier since we could focus on the editorial and spec-factoring issues, assured by test cases that the behavior was correct.

Ah, gotcha. That makes sense. I'll try to modify the example you've provided.

@domenic it looks like your example doesn't have any tests yet, and since it's using export syntax, it won't run without a transpiler in any engine. I assume that making it run in node 6 would be sufficient?

I've added https://github.com/tc39/proposal-promise-finally/tree/strict_tests/especially - node especially/test should equal cat especially/expected, to start.
What tests can I add that would be sufficient? (I can convert the "test" file to be a bunch of assertions, such that it returns zero)

I think just basic smoke tests for the various finally cases would be what I'd hope for:

onFinally throws an exception
onFinally returns a non-promise
onFinally returns a pending-forever promise
onFinally returns an immediately-fulfilled promise
onFinally returns an immediately-rejected promise
onFinally returns a fulfilled-after-a-second promise
onFinally returns a rejected-after-a-second promise

Also it would be good to ensure that the A+ tests still pass even with the modifications made to the various existing promise abstract ops.

Closed with #8.


Howling Moon

The most iconic Full Moon related mob.. The Werewolf. Climbing walls, running fast and jumping extra high, these mobs have 0 issues scaling your defenses and have even less issues eating you for dinner.
Werewolves claws are incredibly strong and can be used to make a set of gauntlets.
equipped via curios in the hand slot. It could allow for climbing walls.

There will be a new wolf mob in the new version of the mod, so Ill see if I can merge some of the features suggested here into it. A gadget for climbing could be interesting


feat(codegen): minify whitespace for throw, await and import expressions

part of #7638

#7671  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

We need to figure out what expressions are required to print whitespace and what are not.


Cherry pick for API_LEVEL_1

Description
Cherry picks from:

https://github.com/LedgerHQ/ledger-secure-sdk/pull/188
#291
#424
#436
#439
#430

Changes include

[ ] Bugfix (non-breaking change that solves an issue)
[x] New feature (non-breaking change that adds functionality)
[x] Breaking change (change that is not backwards-compatible and/or changes current functionality)
[ ] Tests
[ ] Documentation
[ ] Other (for changes that might not fit in any category)

Codecov Report
All modified lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (API_LEVEL_1@27161dc). Click here to learn what that means.

Additional details and impacted files
@@              Coverage Diff               @@
##             API_LEVEL_1     #440   +/-   ##
==============================================
  Coverage               ?   85.96%
==============================================
  Files                  ?        8
  Lines                  ?      399
  Branches               ?        0
==============================================
  Hits                   ?      343
  Misses                 ?       56
  Partials               ?        0

Flag
Coverage Δ

unittests
85.96% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


UnicodeDecodeError for user-agent (still)

I also experienced the problem described in #1485 in the field.  Code to reproduce it is below.  I get that someone is trying to hack the site, but still, it shouldn't 500 error.  Running mezzanine 4.0.1.
from django.test import TestCase, Client

class TestBadAgent(TestCase):

    def test_bad_agent(self):
        bad_str = u"\xe2\x28\xa1"
        c = Client(HTTP_USER_AGENT=bad_str)
        c.get('/')

    def test_bad_agent_issue_in_field(self):
        bad_str = b'\xff'
        c = Client(HTTP_USER_AGENT=bad_str)
        c.get('/')

Edited to add a unit test to show the exact path hit in the field... in the second test case, the unicode decode error is handled, but then raises an exception again when checking against known devices...

I'm also getting some error based on Javascript/SQL injection attacks in the HTTP_USER_AGENT field, e.g:
HTTP_USER_AGENT = '}__test|O:21:"JDatabaseDriverMysqli":3:{s:2:"fc";O:17:"JSimplepieFactory":0:{}s:21:"\0\0\0disconnectHandlers";a:1:{i:0;a:2:{i:0;O:9:"SimplePie":5:{s:8:"sanitize";O:20:"JDatabaseDriverMysql":0:{}s:8:"feed_url";s:3230:"eval(base64_decode(<>));JFactory::getConfig();exit";s:19:"cache_name_function";s:6:"assert";s:5:"cache";b:1;s:11:"cache_class";O:20:"JDatabaseDriverMysql":0:{}}i:1;s:4:"init";}}s:13:"\0\0\0connection";b:1;}\xf0\xfd\xfd\xfd'
I expect the smarter move would be to block such requests using the webserver (e.g. nginx) with something like this:
location / {
    if ($http_user_agent ~* JDatabaseDriverMysql) {
        return 403;
    }
}

See also http://stackoverflow.com/questions/19337662/blocking-all-bots-except-a-few-with-nginx for an example blocking specific bots - you could add a string that appears in these attacks to the patterns checked for.
I am planning to add such checks to my nginx.conf template before doing a redeploy.

Glad to fix this in Mezzanine if someone can provide a test case that reproduces it.

The test case by @jenniferlianne raises the exception consistently. I also get several occurrences of the user agent shown by @molokov in my error logs every day (notice \xf0\xfd\xfd\xfd, which is choking Mezzanine).
I would say we have two issues here, one is the occurrence of non-ascii chars in UA strings (which might or might not be malicious in intentions), and then the documented attacks towards Joomla sites with the "JDatabaseDriver" UA (which are definitely malicious, but harmless towards Django/Mezzanine).
I'm +1 on getting rid of the exception and making the UA handling more robust. Blocking at nginx level is a good idea but a separate issue IMO.

The test case by @jenniferlianne raises the exception consistently.

I'm sorry I completely missed that!

Just pushed the fix, thanks @jenniferlianne for the test case.


correct sort order for my dashboard - fixes #1395

Fixes #1395 .
Changes proposed in this PR:

Changed sort_direction argument from symbol to string

please delete issue1395 branch after merge if you can. thanks


Build Overview Page Feature Request

VSTS (DevOps) currently has a feature that displays all of the current builds in progress, those queued and those that had previously completed (in a timeframe) for the given project.
This same functionality in AppCentre would be highly useful at the organisation level with consideration to security.
I sometimes find myself with x amount of Chrome windows open trying to keep on top of what is building.  If you consider that there are potentially multiple branches building at the same time on a single app, it can mean I need quite a few pages open.
It's also a little intense in its current state because the build status for an individual job is buried down around 4 levels ... Org / App / Build / Branches.
Something to think about anyway.
Thanks for listening.
Brad

Hi Brad, thank you for your feature request. My colleague @nilofer will run it from here, however, it helps a lot if this Github issue gets more :+1: reaction.


Let me download my invoices from Amazon

Please explain your idea to make the app better:
Let me download my invoices from Amazon as pdf. That would be great.
App Version Info:
<EMAIL_ADDRESS>BaPUugaNQMudoU4TEccEIw

The idea in general is good, but it is out of the scope of this app. There are existing online services that allow to do such downloads. OnePile can then be used to manage them.


Remove unneeded comparison to double.nan

Fixes #115546
double.nan doesn't equal to anything, even itself. There is a new hint in Dart SDK for that, which needs Flutter code clearance first.
Pre-launch Checklist

[X] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[X] I read the Tree Hygiene wiki page, which explains my responsibilities.
[X] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[X] I signed the CLA.
[X] I listed at least one issue that this PR fixes in the description above.
[ ] I updated/added relevant documentation (doc comments with ///).
[x] I added new tests to check the change I am making, or this PR is test-exempt.
[ ] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

test-exempt: fixing invariant check

I'm guessing someone else applied this same diff? When this was merged it was an empty commit: https://github.com/flutter/flutter/commit/24865ad83be9daf2bc08e42106417819fa6037b5

Yes, it was https://github.com/flutter/flutter/commit/de4c0b19a2604e2cdf8e662b8e0b0ee28fc66ba1


Inconsistent precedence in Markdown parser

There seems to be a conflict between my custom inline parser and image tags.
This is my code:
const copyOpen = '[copy]';
const copyClose = '[/copy]';
export const CopyExtension: MarkdownConfig = {
    defineNodes: [NodeTypes.copy, NodeTypes.copyTag],
    parseInline: [{
        name: NodeTypes.copy,
        parse(cx: InlineContext, next: number, pos: number) {
            const isOpenTag = cx.slice(pos, pos + copyOpen.length) === copyOpen;
            const isCloseTag = cx.slice(pos, pos + copyClose.length) === copyClose;

            if (isOpenTag) {
                return cx.addDelimiter(CopyDelim, pos, pos + copyOpen.length, true, false);
            } else if (isCloseTag) {
                return cx.addDelimiter(CopyDelim, pos, pos + copyClose.length, false, true);
            }
            return -1;
        },
        after: NodeTypes.italic,
    }],
    props: [
        styleTags({
            [NodeTypes.copyTag]: tags.special(CustomTags.CopyTag),
            [NodeTypes.copy]: tags.special(CustomTags.Copy),
        }),
    ],
};

This issue is that my copy tag gets mistaken for on image when placing a ! character before the closing tag.
Since it is parsed after Italic it should be parsed before images. If I set it to be before image it gets interpreted as a link.
I don't see what else can be done on my end to fix this.
Here are some screenshots of the behaviour:

The order in which inline parsers get applied applies to the first character they match. So at the !, all inline parsers would be tried, in order, and since your custom syntax doesn't match there yet, even if it has a higher precedence than images, it won't match yet, but then the image parser will match, and consume the bracket syntax. As such, you may not be able to parse this notation in combination with images using @lezer/markdown, at the moment.


25:24-31 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted

I use the newest angular cli to get an angular 4 default app. When I install TypedJson with its typings, I get the following errors:
25:24-31 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted
Uncaught Error: Cannot find module "."
at webpackMissingModule (vendor.bundle.js:68185) []
at METADATA_FIELD_KEY (vendor.bundle.js:68185) []
at Object.slashes (vendor.bundle.js:68193) []
at webpack_require (inline.bundle.js:53) []
at Object.167 (main.bundle.js:46) []
at webpack_require (inline.bundle.js:53) []
at Object.168 (main.bundle.js:118) []
at webpack_require (inline.bundle.js:53) []
at Object.160 (main.bundle.js:24) []
at webpack_require (inline.bundle.js:53) []
at Object.399 (main.bundle.js:215) []
at webpack_require (inline.bundle.js:53) []
at webpackJsonpCallback (inline.bundle.js:24) []
at :4200/main.bundle.js:1:1 []
So it seems it not possible to use TypedJson with new webpack/angular/angularcli.

I'm getting the same error using webpack.

Fixed it! I had to add the following plugin webpack's configuration file:
plugins: [
    new webpack.ContextReplacementPlugin(/typedjson-npm/, 'typed-json.js')
]

@inad9300  this is not fixing my issue. is there something else we need to do ?


METRON-1289: Alert fields are lost when a MetaAlert is created

Contributor Comments
This PR fixes several bugs in the ElasticsearchMetaAlertDao that incorrectly update the included alerts.  The MetaAlert endpoints (and underlying DAO interface methods) were refactored to make alert/metaAlert updates easier to implement and understand.  Changes include:

Updates to meta alert alert and status fields are no longer allowed via the http://node1:8082/swagger-ui.html#!/update-controller/patchUsingPATCH endpoint.  There changes require updates to multiple objects and require special handling.
Adding an alert to a metaAlert is now done through the http://node1:8082/swagger-ui.html#!/meta-alert-controller/addAlertsToMetaAlertUsingPOST endpoint and requires only guids (and optional sensor types)
Removing an alert from a meta alert is now done through the http://node1:8082/swagger-ui.html#!/meta-alert-controller/removeAlertsFromMetaAlertUsingPOST endpoint and requires only guids (and optional sensor types)
Updating a meta alert status is now done through a dedicated http://node1:8082/swagger-ui.html#!/meta-alert-controller/updateMetaAlertStatusUsingPOST that requires only a guid and status value

These functions can be tested in full dev as follows:

Create a meta alert with multiple alerts through the http://node1:8082/swagger-ui.html#!/meta-alert-controller/createUsingPOST endpoint and record the meta alert guid that is returned:

{
  "alerts": [
    {
      "guid": "36e0a616-1bf2-409b-a965-123783710595",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
 {
      "guid": "b676dfc4-19b3-4493-9f6a-20690aea25a9",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ],
  "groups": [
    "string"
  ]
}

Verify the meta alert was created correctly with the http://node1:8082/swagger-ui.html#!/search-controller/getLatestUsingPOST endpoint:

{
  "guid": "7398fc8a-6dc5-42dd-b2e0-7e764691eeff",
  "sensorType": "metaalert"
}

The meta alert should contain the 2 alerts in the create request above in the alert field
Verify the meta alert was created correctly by querying ES directly http://node1:9200/metaalert_index/metaalert_doc/7398fc8a-6dc5-42dd-b2e0-7e764691eeff/?pretty (substitutue the meta alert guid you recorded)
Verify the alerts have the metaalerts field populated with the meta alert guid you recorded using findOne and directly against ES (same as last 2 steps with alert guid instead)
Add a new alert with the http://node1:8082/swagger-ui.html#!/meta-alert-controller/addAlertsToMetaAlertUsingPOST endpoint:

{
  "metaAlertGuid": "7398fc8a-6dc5-42dd-b2e0-7e764691eeff",
  "alerts": [
    {
      "guid": "a6f682b0-dea1-4220-9d7f-3829bf54cb06",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ]
}

Repeat steps 2-5 to verify the alert was added to the meta alert and the added alert has the meta alert guid in it's metaalerts field
Remove an alert with the http://node1:8082/swagger-ui.html#!/meta-alert-controller/removeAlertsFromMetaAlertUsingPOST endpoint:

{
  "metaAlertGuid": "7398fc8a-6dc5-42dd-b2e0-7e764691eeff",
  "alerts": [
    {
      "guid": "36e0a616-1bf2-409b-a965-123783710595",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ]
}

Repeat steps 2-5 to verify the alert was removed from the meta alert and the removed alert no longer has the meta alert in it's metaalerts field
Change the meta alert status to "inactive" with the http://node1:8082/swagger-ui.html#!/meta-alert-controller/updateMetaAlertStatusUsingPOST endpoint using the meta alert guid and the value "inactive"
Repeat steps 2-5 to verify both alerts no longer have the meta alert guid in the metaalerts field and the meta alert status has been updated to "inactive".
Change the meta alert status back to "active" using the same endpoint as the previous step
Repeat steps 2-5 to verify the alerts and meta alert look the same as before the status was changed to "inactive"
Try to update the list of alerts (alert field) or status field in a meta alert with a patch request (http://node1:8082/swagger-ui.html#!/update-controller/patchUsingPATCH).  Either case should throw an error that contains an obvious message stating that these fields cannot be directly updated
Updates to any other meta alert field (name, comments, etc) through the http://node1:8082/swagger-ui.html#!/update-controller/patchUsingPATCH endpoint should work normally.

There are several other cases that should be tested as well:

try to add an alert that already exists in a meta alert
try to remove an alert that does not exist in a meta alert
try changing the status from "active" to "active" or "inactive" to "inactive"
repeat the various tests using more than 10 (ES's default search result size) alerts or meta alerts
search should properly include meta alerts when a query matches or exclude alerts that are in a meta alert
any more I'm not thinking of?

The tests described here are implemented in the ElasticsearchMetaAlertIntegrationTest class.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && build_utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

I have tested the rest-api I am able to see the alert details for all alerts under meta alerts
I see a couple of new issues:

When a meta alert is created the meta alert is returned as part of the group api which was not the case before.
When a meta alert is deleted (status set to inactive) the meta alert gets deleted but the details are missing for all alerts under meta alert. I feel this bug might be in there all along but never surfaced since the details were missing when the meta alert is created. It would be nice if you can handle this in this PR.

Do we know why the partial update doesn't work?  I'm not necessarily opposed to doing this as a short term fix, but I'd like to know root cause.
It seems like overkill to submit a full object update on every alert that goes into a metaalert (e.g. that could be a hundred+ full document reindexes happening every time)

@merrimanr For the findAllLatest, can we add a default implementation to the interface that would cover HBase (i.e. just do a for loop lookup), even if it's not efficient.  Otherwise, it should just be a MultiGet right if we want to do something more efficient / HBase specific, right?

No problem.  I am still actively working on this PR and will add the HBaseDao.findAllLatest implementation task to my list.

The 2nd most recent commits updates the MetaAlertDao interface by adding additional methods for interacting with metaalerts.   This should make it easier to use and make the underlying implementation simpler and easier to understand by eliminating the need for complex logic that must catch all of the different use cases in the MetaAlertDao.update.  The high-level changes include:

dedicated methods that allow adding/removing alerts to/from a metaalert
dedicated method for changing the status of a metaalert
direct updates through IndexDao.update are no longer allowed on metaalerts, the other MetaAlertDao methods must be used instead

The latest commit updates the ElasticsearchMetaAlertIntegrationTest with several changes:

test data setup and test style is now consistent across all tests
test cases for the new interface methods were added
old test cases were reconciled with the new test cases
test coverage for ElasticsearchMetaAlertDao is now close to 100% with tests added for all the bugs found so far

I am still working on adding these changes:

updated javadocs and comments in all relevant classes and tests
an HBaseDao.getAllLatest implementation and test
updated ElasticsearchMetaAlertDaoTest adapted for the new MetaAlertDao interface (it's currently all commented out for the purpose of running the integration tests)

For now reviewers can start looking at the new MetaAlertDao interface methods and the accompanying integration tests in ElasticsearchMetaAlertIntegrationTest.  Hope to have the rest pushed out soon.

The latest commits address some outstanding tasks and address a couple other issues:

disabling updates to meta alert objects means we can't update a meta alert name or comments (or other fields that may be added in the future).  The patch implementation was added to the ElasticsearchMetaAlertDao to allow updates to fields other than alert and status which require special handling and have dedicated endpoints
a getAllLatest implementation was added to the HBaseDao along with an integration test
a bug was discovered where the number of search results returned defaults to 10 meaning changes to alerts are only propagated to the first 10 meta alerts and vice-versa.  This was corrected in the ElasticsearchDao.searchByGuids by explicitly setting the size and fixed in the alert/meta alert lookup queries by paging through all the results if necessary.  A test was added to the integration test to catch this case in the future.

I feel like this PR is in pretty good shape from a functional perspective.  Still working on javadocs/documentation but it's ready for functional testing/code review.

@merrimanr Can you merge master into this PR? There is at least one fix in here that caused me problems spinning this up that's in master, so I'd like to have it pulled in so testing is as stable as possible.

whoops, nevermind, I see you already merged master.  my bad. :)

I've spun this up, and add/alert throws this exception:
{
  "responseCode": 500,
  "message": "class org.apache.metron.elasticsearch.dao.ElasticsearchDao: ElasticsearchDao upsert failed: failure in bulk execution:
[1]: index [metaalert_index], type [null_doc], id [be44748e-6471-465f-bd08-7f2c164b6755], message [java.lang.IllegalArgumentException: object mapping [alert] can't be changed from nested to non-nested]
java.io.IOException: ElasticsearchDao upsert failed: failure in bulk execution:
[1]: index [metaalert_index], type [null_doc], id [be44748e-6471-465f-bd08-7f2c164b6755], message [java.lang.IllegalArgumentException: object mapping [alert] can't be changed from nested to non-nested]
	at org.apache.metron.elasticsearch.dao.ElasticsearchDao.batchUpdate(ElasticsearchDao.java:368)
	at org.apache.metron.indexing.dao.MultiIndexDao.lambda$batchUpdate$3(MultiIndexDao.java:79)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
	at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
	at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
",
  "fullMessage": "IOException: class org.apache.metron.elasticsearch.dao.ElasticsearchDao: ElasticsearchDao upsert failed: failure in bulk execution:
[1]: index [metaalert_index], type [null_doc], id [be44748e-6471-465f-bd08-7f2c164b6755], message [java.lang.IllegalArgumentException: object mapping [alert] can't be changed from nested to non-nested]
java.io.IOException: ElasticsearchDao upsert failed: failure in bulk execution:
[1]: index [metaalert_index], type [null_doc], id [be44748e-6471-465f-bd08-7f2c164b6755], message [java.lang.IllegalArgumentException: object mapping [alert] can't be changed from nested to non-nested]
	at org.apache.metron.elasticsearch.dao.ElasticsearchDao.batchUpdate(ElasticsearchDao.java:368)
	at org.apache.metron.indexing.dao.MultiIndexDao.lambda$batchUpdate$3(MultiIndexDao.java:79)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
	at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
	at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
"
}

Request
{
  "alertGuids": [
"9b2e9b29-41a5-45fc-aabe-fd49c63af943",
"0fbea2b2-c4f3-43a8-a717-8ceb3467ae10",
"698289ca-43cf-4add-bf35-2a391ae1c244"
],
  "metaAlertGuid": "be44748e-6471-465f-bd08-7f2c164b6755",
  "sensorTypes": [
"bro"
]
}

I'll dig into it a bit, but I assume something isn't quite being constructed properly.

The latest commit fixes the bug in the previous comment.  The root cause was that HBaseDao.getAllLatest was not properly returning the sensor type and causing subsequent updates in Elasticsearch to fail.  The solution was simple:  look up the sensor type from the document and add it to the return Document.  I added a constant for this field (essentially a copy of ElasticsearchMetaAlertDao.SOURCE_TYPE) to the HBaseDao in anticipation of the ES 5 upgrade where it will no longer be required to change .'s to :'s.  At that point we can just change the constant to Constants.SENSOR_TYPE and it should continue to function.
There are a couple other solutions to this problem that I can think of:

change the getAllLatest interface to include guid/sensorType mappings instead of separate guid and sensorType lists
add a column family to store the sensor type
other more complicated ways of getting the correct sensor type field name

I felt this temporary constant was the simplest but feel free to give opinions on other options.  I also update the HBaseDaoIntegrationTest to expect this field to be returned.
The PR description has been updated to reflect the various interface changes and new testing procedure.

I also found what I believe are 2 fairly significant architectural issues with our DAO abstraction:

https://issues.apache.org/jira/browse/METRON-1314
https://issues.apache.org/jira/browse/METRON-1315

@merrimanr @justinleet tested the new API's they are working fine the only issue I see is

When a meta alert is created the meta alert is returned as part of the group api which was not the case before.

@iraghumitra Do you have the specific metaalerts and calls that you made?  I'm spinning this up again, but it'll be a bit before I can test something myself.

@iraghumitra I'm unable to duplicate the grouping on the current code (although I admittedly ran through a pretty basic example). Here's what I did, so let me know if I missed something, or you have a particular query you made.
Find an alert
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "fields": [
    "guid"
  ],
  "from": 0,
  "indices": [
    "snort"
  ],
  "query": "ip_dst_addr:<IP_ADDRESS>",
  "size": 1
}' 'http://node1:8082/api/v1/search/search'

In this case, I just grabbed one:
"d29925a7-da80-49d0-b7b5-0663380d526f":"snort_index_2<IP_ADDRESS>"

Retrieve that alert from findOne to get something to group by
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "d29925a7-da80-49d0-b7b5-0663380d526f",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

I'll group, arbitrarily, with:
"ip_dst_addr": "<IP_ADDRESS>"

Run a group by
/api/v1/search/group

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "groups": [
    {
      "field": "ip_dst_addr"
    }
  ],
  "indices": [
    "snort",
    "metaalert"
  ],
  "query":"ip_dst_addr:<IP_ADDRESS>"
}' 'http://node1:8082/api/v1/search/group'

Results in:
{
  "groupedBy": "ip_dst_addr",
  "groupResults": [
    {
      "key": "<IP_ADDRESS>",
      "total": 1299,
      "score": null
    }
  ]
}

Create a metaalert with that GUID
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "groups": [
    "test"
  ],
  "guidToIndices": {
"adf65dbb-73fd-4347-a61d-a990166fbbb1":"snort_index_2<IP_ADDRESS>"
}
}' 'http://node1:8082/api/v1/metaalert/create'

It returns the guid, in this case:
715538d9-7f4f-4316-b149-a9659df06de4

(Optionally) Look up the metaalert to double check it was created as expected.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "715538d9-7f4f-4316-b149-a9659df06de4",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Run the group by from before
/api/v1/search/group
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "groups": [
    {
      "field": "ip_dst_addr"
    }
  ],
  "indices": [
    "snort",
    "metaalert"
  ],
  "query":"ip_dst_addr:<IP_ADDRESS>"
}' 'http://node1:8082/api/v1/search/group'

Validate that the count has gone down by one (since the metaalert doesn't return and the child alert is now hidden:
{
  "groupedBy": "ip_dst_addr",
  "groupResults": [
    {
      "key": "3232252537",
      "total": 1298,
      "score": null
    }
  ]
}

@justinleet looking at your results I would expect the count to go down by 2 instead of 1:  1 for the alert that was added to the meta alert and another for the meta alert itself since they shouldn't be included in group by queries.  I think the bug @iraghumitra reported is valid.

The latest commit fixes the group by bug found by @iraghumitra and addresses the feedback above.
The group by fix was simple.  We are now just excluding the "metaalert" sensor type from group by queries.
Refactoring the HBaseDao was more involved.  One of the side effects of storing the sensorType in the row key is that now we require a map of guids to sensorTypes for an IndexDao.getAllLatest call.  These means the IndexDao.getAllLatest interface needed to change.  It made sense to just create a standard interface that can be used for looking up records since it is used in many different places and can be confusing as to what is needed to perform an operation on a document.  So this PR introduces a GetRequest that has the following structure:
{
  "guid": "some guid",
  "sensorType" : " some sensor type",
  "index": "some optional index, not required"
}

Since now keep the sensorType in the row key it is required for a look up.  HBase still doesn't have any concept of an index so that is kept optional.  In the ElasticsearchDao this is handled by using the index of a GetRequest if present or looking it up in Elasticsearch if not.  Now it should be clear what is needed to get a document since the interface is more consistent across that DAO methods.
Because of the GetRequest change the testing process and REST interface has changed slightly.  The PR description has been updated to reflect the changes.  I am still planning on doing a fresh vagrant build but this should be ready for testing.

+1 for functionality all the API's are working fine. Great job @merrimanr  👍

Double check me on that logic though. I could definitely be masking an off by one error there.

Adding alerts and adding a preexisting alert
Find two alerts
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "fields": [
    "guid"
  ],
  "from": 0,
  "indices": [
    "snort"
  ],
  "query": "ip_dst_addr:<IP_ADDRESS>",
  "size": 2
}' 'http://node1:8082/api/v1/search/search'

Results in two guids:
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d

Create a metaalert with only one of the alerts
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ],
  "groups": [
    "test"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'

Make sure to get the resulting guid from the response.
6a4affe4-02ce-4d25-80b1-bfc4ca53f557

Retrieve the meta alert and ensure it contains the provided alert
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Retrieve the child alert and ensure the 'metaalerts' field contains the new GUID of the new metaalert
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

Add the same alert to the meta alert
/api/v1/metaalert/add/alert

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ],
  "metaAlertGuid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557"
}' 'http://node1:8082/api/v1/metaalert/add/alert'

It should return "false" as no alerts have been added.  The meta alert should be retrieved again to validate.
Run the add alert again but with the second alert
/api/v1/metaalert/add/alert

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
     {
      "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ],
  "metaAlertGuid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557"
}' 'http://node1:8082/api/v1/metaalert/add/alert'

It should return true, because the second alert will be added.  The meta alert should be retrieved again to validate.
Retrieve the child alerts
Ensure they have the 'metaalerts' field populated with their parent.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

Removing alerts and removing an already removed alert
Find two alerts
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "fields": [
    "guid"
  ],
  "from": 0,
  "indices": [
    "snort"
  ],
  "query": "ip_dst_addr:<IP_ADDRESS>",
  "size": 2
}' 'http://node1:8082/api/v1/search/search'

Results in two guids:
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d

Create a metaalert with the alerts
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
     {
      "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }

  ],
  "groups": [
    "test"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'

Make sure to get the resulting guid from the response.
b25b663e-39c9-42d5-a52c-e6380235d43f

Retrieve the meta alert and ensure it contains the provided alerts
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Remove one of the alerts
/api/v1/metaalert/remove/alert

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ],
  "metaAlertGuid": "b25b663e-39c9-42d5-a52c-e6380235d43f"
}' 'http://node1:8082/api/v1/metaalert/remove/alert'

Retrieve the meta alert again, and ensure it only contains the second alert.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Rerun the delete
/api/v1/metaalert/remove/alert

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ],
  "metaAlertGuid": "b25b663e-39c9-42d5-a52c-e6380235d43f"
}' 'http://node1:8082/api/v1/metaalert/remove/alert'

Retrieve the meta alert again, and ensure it only contains the second alert.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Retrieve the child alerts
Ensure only the second alert has the 'metaalerts' field populated with the parent met alert.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

Changing Metaalert status
Find two alerts
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "fields": [
    "guid"
  ],
  "from": 0,
  "indices": [
    "snort"
  ],
  "query": "ip_dst_addr:<IP_ADDRESS>",
  "size": 2
}' 'http://node1:8082/api/v1/search/search'

Results in two guids:
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d

Create a metaalert with the alerts
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
     {
      "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }

  ],
  "groups": [
    "test"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'

Make sure to get the resulting guid from the response.
da60ccc9-9e79-45c5-be07-0a322c8791f0

Retrieve the meta alert and ensure it contains the provided alerts
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Change the meta alert status to active
This makes sure nothing happens when we set active status to the same active status.
/api/v1/metaalert/update/status/{guid}/{status}

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' 'http://node1:8082/api/v1/metaalert/update/status/da60ccc9-9e79-45c5-be07-0a322c8791f0/active'

It should return false, as no status has changed.
Retrieve the metaalert and ensure it is still active
Look for the 'status' field.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Retrieve the child alerts
Ensure both alerts have the 'metaalerts' field populated with the parent meta alert.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

Change the meta alert status to inactive
Look for the 'status' field.  Running this once will set it to 'inactive'.  Subsequent runs have no effect ('inactive' -> 'inactive' does nothing).
/api/v1/metaalert/update/status/{guid}/{status}

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' 'http://node1:8082/api/v1/metaalert/update/status/da60ccc9-9e79-45c5-be07-0a322c8791f0/inactive'

It should return true, because the status has changed.
Retrieve the metaalert and ensure it is inactive
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Retrieve the child alerts
Ensure neither alert has the 'metaalerts' field populated with the parent meta alert.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

Change status back to active
/api/v1/metaalert/update/status/{guid}/{status}

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' 'http://node1:8082/api/v1/metaalert/update/status/da60ccc9-9e79-45c5-be07-0a322c8791f0/active'

It should return false, as no status has changed.
Retrieve the metaalert and ensure it is active again
Look for the 'status' field.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Retrieve the child alerts
Ensure both alerts have the 'metaalerts' field populated with the parent meta alert.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

Create meta alert with more than 10 alerts
Find more than 10 alerts alerts
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "fields": [
    "guid"
  ],
  "from": 0,
  "indices": [
    "snort"
  ],
  "query": "ip_dst_addr:<IP_ADDRESS>",
  "size": 11
}' 'http://node1:8082/api/v1/search/search'

Note the alerts that come back
62a53a5f-78e6-417a-8078-fb850baa3e84
876b72cb-9d72-4706-ac99-46cf91a8f359
5fd8b0a0-1f68-494a-ae20-633542a7045d
aee597a0-4255-499a-a4e2-ec7d756babb2
bf9e0e73-e64c-4759-b4f7-efad0a60be82

5ab9ce98-30db-45b2-a4e6-6489f136c839
0a4a7019-04f8-4a8c-af0b-d2e3908ecdc9
3423fdca-cefa-402a-b57d-60b75a15f046
2eb63002-e5f2-467a-8675-30b653ae145b
53f38cfd-aa89-4e49-ba5f-827eb73774cd

5f71a515-4976-4b0d-be85-bb6879b1e151

Create a metaalert with the alerts
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "62a53a5f-78e6-417a-8078-fb850baa3e84",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
     {
      "guid":"876b72cb-9d72-4706-ac99-46cf91a8f359",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
    {
      "guid": "5fd8b0a0-1f68-494a-ae20-633542a7045d",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
    {
      "guid": "aee597a0-4255-499a-a4e2-ec7d756babb2",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
     {
      "guid":"bf9e0e73-e64c-4759-b4f7-efad0a60be82",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
    {
      "guid": "5ab9ce98-30db-45b2-a4e6-6489f136c839",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
     {
      "guid":"0a4a7019-04f8-4a8c-af0b-d2e3908ecdc9",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
    {
      "guid": "3423fdca-cefa-402a-b57d-60b75a15f046",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
    {
      "guid": "2eb63002-e5f2-467a-8675-30b653ae145b",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
     {
      "guid":"53f38cfd-aa89-4e49-ba5f-827eb73774cd",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    },
     {
      "guid":"5f71a515-4976-4b0d-be85-bb6879b1e151",
      "index": "snort_index_2<IP_ADDRESS>",
      "sensorType": "snort"
    }
  ],
  "groups": [
    "test"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'

Make sure to get the resulting guid from the response.
00eae5ba-6137-4601-ae3a-fbf0003e58e6

Retrieve the meta alert and ensure it contains the provided alerts
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Retrieve the child alerts
Ensure all alerts have the 'metaalerts' field populated with the parent meta alert.
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid":"62a53a5f-78e6-417a-8078-fb850baa3e84",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

... // 10 more times

Patch alert and status
Create a metaalerts and get the GUID for the following steps.
Attempt to update status field
/api/v1/update/patch

curl -X PATCH --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "patch": [
    {
      "op": "replace"
        , "path": "/status"
        , "value": "failure"
    }
  ],
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'

Should return
{
  "responseCode": 500,
  "message": "Meta alert patches are not allowed for /alert or /status paths.  Please use the add/remove alert or update status functions instead.",
  "fullMessage": "IllegalArgumentException: Meta alert patches are not allowed for /alert or /status paths.  Please use the add/remove alert or update status functions instead."
}

Attempt to update the alert list
/api/v1/update/patch

curl -X PATCH --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "patch": [
    {
      "op": "replace"
        , "path": "/alert"
        , "value": [{
            "alertOne":"test"
        },
        {
             "alertTwo":"test"
        }
        ]
    }
  ],
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'

Should return
{
  "responseCode": 500,
  "message": "Meta alert patches are not allowed for /alert or /status paths.  Please use the add/remove alert or update status functions instead.",
  "fullMessage": "IllegalArgumentException: Meta alert patches are not allowed for /alert or /status paths.  Please use the add/remove alert or update status functions instead."
}

Patch neither alert and status
Create a metaalerts and get the GUID for the following steps.
Patch in new field
/api/v1/update/patch

curl -X PATCH --header 'Content-Type: application/json' --header 'Accept: */*' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "patch": [
    {
      "op": "add"
        , "path": "/name"
        , "value": "My new meta alert name"
    }
  ],
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'

Retrieve the meta alert and ensure it contains the new 'name' field
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Code-wise, I'm pretty good at this point.  Once the docs come in, I'll give them a once-over and hopefully we're good to go soon.
Thanks a lot for the hard work here!

@justinleet same here.  My +1 is imminent pending docs.

This looks good to me.  I will not require attribution to my PR against this branch and I give this a +1 pending travis.

+1, looks good. Thanks for all the work on the supplemental fixes. Feel free to skip attribution on the testing PR.


Read switches from file

It would be a nice addition that I saw already somewhere: if there is an item starting with a @ on the command line, it's considered to be a config file to contain switches and its contents are read and included. I added it for myself with a very small modification, although it might be worked into the whole package in a nicer way for sure:
in InstanceBuilder.Build(), locate the
var argumentsList = arguments.Memorize();

line (around line 48), and replace it with:
  var argumentsList = new List<string>();
  foreach (string argument in arguments) {
    if (argument[0] == '@') {
      string filename = argument.Substring(1);
      string switches = string.Join(" ", File.ReadAllLines(filename));
      argumentsList.AddRange(switches.Split(new[] { ' ' }));
    }
    else
      argumentsList.Add(argument);
  }

Some more error checking would be required, too, of course.

Good idea. There are some considerations:

Value options that start with "@" can no longer be used, because it will be interpreted by the parser as filename.
This can cause problems to the applications already running and using "@" in start of options when upgrade to the new release.
Help should be modified to support  this feature under the control of some setting  to show/hide file parameter usage.
Error handling should be modified to handle file errors(file not exist, file read error,...)

I thinks the feature can be added as an Extension method without the change in the Parser Code base.
Developers, in the application side, manage the above issues  using custom help.
The Extension method, like:
  public static class CommandLineExtensions
  {
     public static string[] ExpandArgs(this string[] arguments)
     {
        //rest of code
        //Handle file errors
      }
   }

Usage:
 result = Parser.Default.ParseArguments<Options>(args.ExpandArgs());

What do you think @ericnewton76?

I just like to open the same request.
@moh-hassan
i agree to your considerations…
But i see no way to handle the Errors or reflect the expansion in Help when you do it this way.
From my point, this should be really done by only modify the settings, so the the two overwrites of the Parser.ParseArguments methodes are the best (from my view) places to add this logic.
Emmo

@motwok
With the extension method solution, developer can create a custom help in his application and add something like:

Note: If you start the options  with the letter @, the rest should be a filename of switches

Also, he can handle the errors of file (file not exist,..)
So we can add  Extension methods as an extensibility mechanism to the CommandLineParser.
In case of adding  the feature as built-in method in the Parser and controlled by setting in the ParserSettings, help and errors are managed based on this setting.
Is it right?

@moh-hassan
sure… but i think this Expansion should be integrated… if you have it outside of Parser you "must" do all the the things manually. Sounds like reinventing the wheel for me.

Yes, I mean that the extension method is merged and be a part of the package.
Reinventing the wheel should be avoided.
As you prefer that the feature be part of the Parser, no problem for me.
@deakjahn, can submit PR.


Report a bug

Report a bug:
Information
Using:
Allsigns v1.0.10
PocketMine-MP verison: 3.0.2
API: 3.0.0
PHP: 7.2
Bug Description
When I run the server,it happen an error and disable this plugin.
Debug
[13:16:03] [Server thread/INFO]: Enabling AllSigns v1.0.10
[13:16:03] [Server thread/CRITICAL]: Error: "Call to undefined method pocketmine\Server::getScheduler()" (EXCEPTION) in "AllSigns.phar/src/surva/allsigns/AllSigns" at line 27
[13:16:03] [Server thread/DEBUG]: #0 src/pocketmine/plugin/PluginBase(102): surva\allsigns\AllSigns->onEnable()
[13:16:03] [Server thread/DEBUG]: #1 src/pocketmine/plugin/PluginManager(618): pocketmine\plugin\PluginBase->setEnabled(boolean 1)
[13:16:03] [Server thread/DEBUG]: #2 src/pocketmine/Server(1928): pocketmine\plugin\PluginManager->enablePlugin(surva\allsigns\AllSigns object)
[13:16:03] [Server thread/DEBUG]: #3 src/pocketmine/Server(1914): pocketmine\Server->enablePlugin(surva\allsigns\AllSigns object)
[13:16:03] [Server thread/DEBUG]: #4 src/pocketmine/Server(1708): pocketmine\Server->enablePlugins(integer 1)
[13:16:03] [Server thread/DEBUG]: #5 src/pocketmine/PocketMine(237): pocketmine\Server->__construct(BaseClassLoader object, pocketmine\utils\MainLogger object, string C:\Users\Administrator.USER-20180220MY\Desktop\PocketMine-MP, string C:\Users\Administrator.USER-20180220MY\Desktop\PocketMine-MP\plugins)
[13:16:03] [Server thread/DEBUG]: #6 C:/Users/Administrator.USER-20180220MY/Desktop/PocketMine-MP/PocketMine-MP.phar(1): require(string phar://C:/Users/Administrator.USER-20180220MY/Desktop/PocketMine-MP/PocketMine-MP.phar/src/pocketmine/PocketMine.php)
[13:16:03] [Server thread/INFO]: Disabling AllSigns v1.0.10
Please also do:
Ask me if you need more infomation.
Please help to fix the bug.
When the bug fixed,please comment below to tell me and I will close this issue
Thank you for helping.Have a good day:D

Fixed.Closing Issue.

Fixed.Closing Issue.


Add option to replicate pager on issue page also on the top

You now have to scroll all the time. With a 130% page zoom, I get around 8 issues per screen. I have to scroll a bit to get to the pager and it gets tiring if I want to visit every page and do Ctrl-F to search for a specific word in the issue titles.

I don't think that would look good. Would be better to add keyboard shortcuts for going to next/previous issue page.

I have to scroll a bit to get to the pager and it gets tiring if I want to visit every page and do Ctrl-F to search for a specific word in the issue titles.

Why are you not using the actual search? For example, in:title foo

I didn't know about in:title, thanks. Still, shortcuts would be really handy. How can I add them? Is there an option somewhere?

It has to be added to Refined GitHub as a feature, ala https://github.com/sindresorhus/refined-github/blob/master/source/features/add-create-release-shortcut.js

I'm on it if it's okay, @tterranigma.

There’s also “cmd+down” (OS X) or “end” key (Windows) to scroll to the bottom of the page


Why is the  --keep-geno argument given to vcfallelicprimitives? It leads to invalid vcf files?

Hi,
Why is the  --keep-geno argument given to vcfallelicprimitives?
The vcfallelicprimitives specifies that this is not the default behavior and that it  could lead to invalid vcf files.
-g, --keep-geno         Maintain genotype-level annotations when decomposing.  Similar
caution should be used for this as for --keep-info.
If I post process a vcf file using the same post processing tools / commands as bcbio does  I get an invalid vcf file.
https://github.com/chapmanb/bcbio-nextgen/blob/master/bcbio/variation/freebayes.py#L108
The error I get by using vcf-validator from vcftools is:
Sample_03 at Chr_01:2068400 .. FORMAT tag [GL] expected different number of values (expected 3, found 10),FORMAT tag [AO] expected different number of values (expected 1, found 3),FORMAT tag [QA] expected different number of values (expected 1, found 3)
And this seems to be related to that the GL, AO and QA fields of the samples don't correspond anymore to the number of alternative alleles present in the ALT column.
The original variant was
Chr_01  2068400        .       ATGTTTTTC       CTATTGTTT,CTATTGTTC,ATATTGTTC
and after vcfallelicprimitives the variant is
Chr_01  2068400        .       A       C
but the genotypes GL, AO and QA values don't correspond anymore.
Or does bcbio do something more downstream to bring the genotype values in correspondence with the variant alleles again?

Thanks for the details writeup about the issues you're running into. We use --keep-geno as an imperfect choice when trying to decompose larger variants. Without it we lose all genotype annotations, which isn't great, and with it you have the possibility of getting incorrect decomposition. We haven't been running into issues with the bcbio output so happy to try and dig into this.
Are the problem variants you ran into out of bcbio, or a separate custom pipeline? Are downstream tools complaining about them, or were you just checking the VCF with vcf-validator? Sometimes with VCF it's worth being practical as trying to bring everything into perfect spec can be a lot of work. If you can put together an example of an input and problem output we could try to dig more. Hope this helps.

For now I ran the Freebayes command and the post processing steps outside of bcbio.
The original variant and the first two samples were
Chr_01 2068400 . ATGTTTTTC CTATTGTTT,CTATTGTTC,ATATTGTTC
GT:GQ:DP:RO:QR:AO:QA:GL   .
1/3:1.26014e-09:22:4:148:9,4,4:332,147,148:-10,0,-10,-10,-0.0911111,-10,-10,0,-10,-10
1/2:135.313:28:3:111:12,9,4:442,320,148:-10,-10,-10,-10,0,-10,-10,-10,-10,-10
After vcfallelicprimitives with --keep-geno the (first) variant is
Chr_01 2068400 . A C
GT:GQ:DP:RO:QR:AO:QA:GL
.:.:.:.:.:.:.:.
1|0:1.26014e-09:22:4:148:9,4,4:332,147,148:-10,0,-10,-10,-0.0911111,-10,-10,0,-10,-10
After vcfallelicprimitives without --keep-geno the (first) variant is
Chr_01  2068400        .       A       C
GT
.
1|0
So the trade made here is that you still have a usable GQ:DP:RO:QR fields, but that the AO:QA:GL fields are some what  invalid after specifying --keep-geno as an argument.
We make use downstream of the AO values (alternative allele counts).
For variants converted to primitives we now can't depend anymore on that the number of alternative alleles is equal to the number of alternative allele counts.
At least I know understand the imperfect choice made and we will see if we can find a work around.
Thanks!

Thanks for the details. If we can find tools that do a better job of this allelic primitive decomposition while maintaining all of the fields correctly I'd be happy to test and sub them in. Glad you can find a workaround downstream and thanks again.

Hi Brad,
Could you please pass the -t argument to vcfallelicprimitives in the freebayes variant calling pipe:
https://github.com/chapmanb/bcbio-nextgen/blob/master/bcbio/variation/freebayes.py#L125
This will tag the decomposed variants and make it easier to handle them correctly downstream.  I guess an info field will be created.
vcfallelicprimitives -h
usage: vcfallelicprimitives [options] [file]

options:
    -m, --use-mnps          Retain MNPs as separate events (default: false).
    -t, --tag-parsed FLAG   Tag records which are split apart of a complex allele with this flag.
    -L, --max-length LEN    Do not manipulate records in which either the ALT or
                            REF is longer than LEN (default: 200).
    -k, --keep-info         Maintain site and allele-level annotations when decomposing.
                            Note that in many cases, such as multisample VCFs, these won't
                            be valid post-decomposition.  For biallelic loci in single-sample
                            VCFs, they should be usable with caution.
    -g, --keep-geno         Maintain genotype-level annotations when decomposing.  Similar
                            caution should be used for this as for --keep-info.

Thank you !


Buffs / Debuffs positon

Now that you polished everything up and it's looking GREAT, the only thing missing is buffs/debuffs position (above - below).
Again, amazing work!

Thank you @mrrosh !
I'll think about making an option for that, but I think most people got other addons for that?
Addons like MoveAnything or VCB?
I'll look into it! 👍

This thread might (?) be useful to make this happen, atleast shirsig linked to something but I have no clue what i'm looking at lol https://forum.elysium-project.org/topic/32153-need-help-to-change-enemy-debuff-position-and-growth-direction-default-unitframes/

@Cruzix Perfect, that's very useful! saved me some time finding the function. 😀

Also, any chance you could make the enemy debuffs clickthrough, so if you accidentally hit them when trying to move the camera, u'd still be able to do so? :)


Whether it can be turned on/off automatically by hovering over the mouse

It is suggested that it can be turned on/off by hovering over the mouse, just like lobehub in A1111

Or, could you add a shortcut to turn on/off the sidebar？

well Alt + Z is opening/closing the sidebar

Now the sidebar will open when you hover on the border, and hide when you click outside!

Now the sidebar will open when you hover on the border, and hide when you click outside!

Great！ thank u very much！


Add decoding/reader support

Currently you can only encode, not decode.

This is underway on the decoder branch.


chore(product): Define High-level Features/Milestones

[x] Define Milestones
[ ] Define High Level Features

Milestones can be seen here
The linked view sorts the milestones by closest due date offering a clear understanding of the progression of tasks leading up to the completion of the final product.


Can't open item details using MS Edge

Clicking / touching / pen-ing the header of an HTML5 feature has no effect.

what the hell, it seems to work now. No console errors in either working scenarios or broken scenarios. I guess a weird Edge bug?


feat: Add use-layers rule

Prerequisites checklist

[x] I have read the contributing guidelines.

What is the purpose of this pull request?
Add a new use-layers rule
What changes did you make? (Give an overview)

Added use-layers rule
Added tests
Added docs
Ensured the rule is exported from the plugin

Note: This rule is not recommended because it will likely only be useful on a subset of CSS files in any given project.
Related Issues
fixes #18

Is there anything you'd like reviewers to focus on?

Should @import require a layer by default? Right now it doesn't, but I feel like maybe it should.
CSSTree is returning the wrong location for layer names. I've opened an issue for that: https://github.com/csstree/csstree/issues/309

Should @import require a layer by default? Right now it doesn't, but I feel like maybe it should.

I think so. I had the same thought before seeing this PR.

@fasttime I can't quite figure out why the type tests are failing now. Can you take a look?

@jeddy3 thanks, updated!

I think I figured out the types issue: https://github.com/eslint/css/pull/28

I fixed the bug in CSS Tree with layer positioning. Just waiting for a new release to update this PR.

With CSSTree updated, all of the locations are now correct. :tada:


loss

ahha, i have a question is about the loss , i  see you set the loss default value is 1*L1, when i use the ADV will error , dataset is the GOPRO_Large,can you help me ?thx

What was the exact command you tried and what kind of error message did you encounter?

Beware that you should always put multiplier in the loss argument
You may need more GPU memory when using adversarial loss if that was an out-of-memory error. In that case, mixed-precision training is recommended. Currently, Apex is required for mixed-precision training. When PyTorch 1.6 is out, I will update the amp part to PyTorch native using torch.cuda.amp.

ok ,the lamda default value is 1e-4 ?dataset  GOPRO_Large

Do you mean \lambda in equation 6?
No, the loss term used for this repository is different and default lambda is 0.
It is up to your choice to put adversarial weights.
As shown in usage examples, you can try several weights.
# adversarial training
python main.py --n_GPUs 1 --batch_size 8 --loss 1*L1+1*ADV
python main.py --n_GPUs 1 --batch_size 8 --loss 1*L1+3*ADV
python main.py --n_GPUs 1 --batch_size 8 --loss 1*L1+0.1*ADV
python launch.py --n_GPUs 2 main.py --batch_size 16 --loss 1*L1+3*ADV

If you need to reduce memory usage, set --amp true.
# adversarial training
python main.py --n_GPUs 1 --batch_size 8  --amp true --loss 1*L1+1*ADV
python main.py --n_GPUs 1 --batch_size 8  --amp true --loss 1*L1+3*ADV
python main.py --n_GPUs 1 --batch_size 8  --amp true --loss 1*L1+0.1*ADV
python launch.py --n_GPUs 2 main.py --batch_size 16 --amp true --loss 1*L1+3*ADV

ok , i got it ,really thank you


Using npm and traditional node, cannot use because of it forcing es6.

Just a heads up. I don't think there is an es5 version being built for npm. So by doing:
npm install --save tvdml

and having a file of:
var TVDML = require('tvdml');

console.log(TVDML);

I get this error:
/Users/firl/Desktop/scratch/testtvdml/node_modules/tvdml/src/index.js:1
(function (exports, require, module, __filename, __dirname) { import * as hooks from './navigation/hooks';
                                                              ^^^^^^
SyntaxError: Unexpected token import
    at Object.exports.runInThisContext (vm.js:76:16)
    at Module._compile (module.js:528:28)
    at Object.Module._extensions..js (module.js:565:10)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)
    at Function.Module._load (module.js:424:3)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/firl/Desktop/scratch/testtvdml/test.js:1:75)
    at Module._compile (module.js:556:32)

This is just a heads up since the README mentions that I should be able to do that. Just started using the library, quite enjoying it.

Thank you! Totally forget to add dist folder to npm package. Will fix it in nearest days ;)

Published v1.0.1 to npm in order to fix this.


Difficult overriding behavior for members added/removed in TeamsActivityHandler

Version
4.13
Describe the bug
Behavior of methods on_teams_members_added and on_teams_member_removed of TeamsActivityHandler are difficult to override due to calling super() definition instead of self.
To Reproduce
Implement a TeamsActivityHandler class which custom behavior in any of those methods
Expected behavior
These methods call the implementation on self.
Additional context
This bug was suggested in the discussion of #1662

@yosshy expect this fix to be released in our next iteration, in the meantime you can build from source if you held a local copy of the libraries or install a nightly build


fix: Use HashMap::from

Summary
Simplifies creating metadata.
Checklist

[x] ran cargo make fmt
[x] using conventional commits to hightlight user-facing fixes and features

Minor request, can we use std::collections::HashMap to reduce the noise a little?


DateTime.UtcNow returning wrong time

I think the deck below (showing that DateTime.UtcNow displays the wrong time) may be a symptom of another problem under the hood.
Given a previously-saved UTC time without timezone information, I believe we can usually reconstruct its .NET representation using:
    // set DateTimeKind to be UTC.
    DateTime myUTCTime = DateTime.SpecifyKind(myTimeWithoutTimeZoneInfo, DateTimeKind.Utc);

The above used to work fine until updating to 16.8.2. Now, comparisons between myUTCTime and DateTime.UTCNow are returning a timezone offset. I noticed the issue below when attempting to reproduce this problem in Deck.
Steps To Reproduce
https://deck.net/9a65e5cb6b9b80fe4f69d1bf9b3c69dd
public class Program
{
    public static void Main()
    {
        Console.WriteLine("UTC time: " + DateTime.UtcNow);
        Console.WriteLine("Local time: " + DateTime.Now);
        Console.WriteLine("Local - UTC: " + (DateTime.Now - DateTime.UtcNow));
    }
}

Expected Result
UTC time: 03/21/2018 09:23:14
Local time: 03/20/2018 14:23:14
Local - UTC: -07:00:00

Actual Result
UTC time: 03/21/2018 04:23:14
Local time: 03/20/2018 14:23:14
Local - UTC: -07:00:00

Hi @yctri – Thanks for the report. We are investigating.
We're looking into the Deck Share feature too. Looks like GitHub changed something with their Gist API that is now causing a failure in Deck.

@yctri – This issue has been fixed and pull-request has been submitted. Once testing and final review are complete, the pr will be merged into v17 and will be included in the Bridge 17.0 release.
The problem was related to the timezoneoffset being added twice to the DateTime object.
If this is a supercritical defect for you, we might be able to make 16.8.3 release, or at least merge this pr into the dev branch (currently 16.8.3) which would allow you to build local packages.
Deck share links are still failing because GitHub changed something in their API, but appears to be undocumented. We have submitted a technical support ticket with GitHub.

This expected result does not make sense to me:
UTC time: 03/21/2018 09:23:14
Local time: 03/20/2018 14:23:14
Local - UTC: -07:00:00

Wouldn't that be, rather,
UTC time: 03/20/2018 21:23:14
Local time: 03/20/2018 14:23:14
Local - UTC: -07:00:00

?

@yctri – shouldn't your UTC offset be +05:00:00 given those UTC vs Local times?

@geoffreymcgill @fabriciomurta Yes, sorry the expected UTC time should have read 21:23:14 (forgot to convert to 24h clock).
What's the ETA on 17.0?

Deck sharing is working again.
Once we know a release date, the due date on the 17.0 milestone will be set. We're working hard on 17.0, and hope to have a -beta available within 2 weeks.

The pull request was merged upstream! The fix will be available on Bridge v17!

The dev.deck instance has been updated and your original sample should now be working correctly.
https://dev.deck.net/9a65e5cb6b9b80fe4f69d1bf9b3c69dd
Hope this helps.

Awesome, thanks.


Environments sometime don't load and clicking 'Create new environment' causes a crash

Dev Home version
0.1701.597.0
Windows build number
10.0.22631.0
Other software
N / A
Steps to reproduce the bug
I haven't got solid steps to repro this every time, but it's happening quite frequently with me visiting the environments page. Don't really know what triggers it. What happens is that sometimes Environments don't load properly, and it shows "No environments found". When I click to "Create new environment", devhome crashes. See the video for better understanding. First time the environments load correctly, but second time they don't.
https://github.com/user-attachments/assets/225f190d-91ce-4089-a5c5-d5828b19a718
Expected result
Environments screen to always load correctly
Actual result
Environments not always loaded, and cause a crash when you click "Create new environment"
Included System Information
Windows: Windows.Desktop v10.0.22631.4037
System Architecture: X64
Included Extensions Information
DevHome GitHub Extension version 0.1700.597.0

/logs

Hi @mdanish-kh, would you be able to file a feedback hub bug for this issue and provide us with the link? That will help us diagnose this issue faster.
You can launch the Feedback hub app by entering the Windows Key + the F key on your keyboard at the same time. During your feedback, there will be an option to recreate the issue in your video. Feedback hub will capture the data needed for further debugging on our side.
Thank you!

@bbonaby Feedback Hub was not cooperating yesterday, so it took me some time to get back. This should be the issue: https://aka.ms/AAs318i
I also used the "Recreate the issue" and was able to successfully repro it. Let me know if you need anything else from me

Fixed as a side effect of #3907


Got everything

Thank you for your feedback!
The issues in this repo are for documentation feedback.
If you are submitting your documentation feedback issue directly through GitHub, please provide the URL of the article so we can route the feedback to the correct author.
Please report Azure DevOps or TFS software bugs, feedback, or questions in the Azure DevOps Developer Community.

@Mannyg0420 Thank you for the feedback.
If you have any further questions or issues with the documentation please comment below.


update excell template to use latest data

Coverage remained the same at 39.163% when pulling c98f70d2259f4a7cd1ce51da64f5f4eabe62be2e on meomancer:update-template into 635322d00a299145fb67a4948a048c3b1002f67f on healthsites:develop.


gh-pages gets overwritten when new tags are pushed

So we typically use github pages for the helm, but we also have github pages for the docs. I suspect these are conflicting a bit.

yeah, the mkdocs deploy --force is overwriting the helm release


Externs for Pixi v3.x rewrite

https://github.com/GoodBoyDigital/pixi.js/issues/1296

alpha released.


mediation not working

I am trying to add Flurry into my Admob mediation. I added Flurry in the admob ad unit as a new ad source, installed the adapter plugin cordova-plugin-admob-flurry, and the rebuilt my app. When I run my app, I notice my banner ad still always come from Admob. I don't see any ad coming from Flurry.
Could you let me know if I set up things correctly? What else I should do besides what I have done above?

Please ask Google for this mediation ad content.
Here is their official forum:
https://groups.google.com/forum/#!forum/google-admob-ads-sdk


Iterate through all derivations in nixpkgs

Is it possible with pythonix to view a derivation similar to "nix show-derivation ..."? I have tried nix.eval("with import  {}; python3Packages.numpy") and ran into errors. Maybe this is not the correct way to view a derivation output? My end goal is that I want to create an adjacency matrix of runtime vs test vs build time python dependencies. But I have plans to use all the derivations to gather statistics. Right now the big limitation for me is how to extract this from nix.

I have implemented this in https://github.com/tweag/pynixutil.
You will still need to instantiate the derivation using the CLI (or pythonix!).


ci: enable publishing, signing, and s3 upload for goreleaser

Release notes included in changes v1.39.1
Large amount of implementation work on release.
Only thing remaining will be to run release/update changelog and work through any final azure pipeline release issues if they occur with the cert/signing.

@andrii-zakurenyi I ran into issues with my local test:

🧪 Test Failure Output
=== Skipped
=== SKIP: auth TestToken_GcpSignJwt (0.00s)
    auth_test.go:527: gcp creds missing, skipping test

=== Failed
=== FAIL: commands TestHandleAuditSearch/Start_date_in_the_future (0.00s)
    audit_test.go:109:
                Error Trace:    /Users/sheldonhull/git/github.com/DelineaXPM/dsv-cli/commands/audit_test.go:109
                Error:          Not equal:
                                expected: &errors.ApiError{stack:[]string{"error: start date cannot be in the future"}, httpResponse:(*http.Response)(nil)}
                                actual  : (*errors.ApiError)(nil)

                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,7 +1,2 @@
                                -(*errors.ApiError)({
                                - stack: ([]string) (len=1) {
                                -  (string) (len=41) "error: start date cannot be in the future"
                                - },
                                - httpResponse: (*http.Response)(<nil>)
                                -})
                                +(*errors.ApiError)(<nil>)

                Test:           TestHandleAuditSearch/Start_date_in_the_future
    --- FAIL: TestHandleAuditSearch/Start_date_in_the_future (0.00s)

=== FAIL: commands TestHandleAuditSearch (0.00s)

=== FAIL: version TestReadCache/Correct_case (0.01s)
    update_test.go:115:
                Error Trace:    /Users/sheldonhull/git/github.com/DelineaXPM/dsv-cli/version/update_test.go:115
                Error:          Not equal:
                                expected: &version.latestInfo{Latest:"1.29.0", Links:map[string]string{"darwin/amd64":"https://dsv.secretsvaultcloud.com/downloads/cli/1.29.0/dsv-darwin-x64"}}
                                actual  : (*version.latestInfo)(nil)

                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,7 +1,2 @@
                                -(*version.latestInfo)({
                                - Latest: (string) (len=6) "1.29.0",
                                - Links: (map[string]string) (len=1) {
                                -  (string) (len=12) "darwin/amd64": (string) (len=69) "https://dsv.secretsvaultcloud.com/downloads/cli/1.29.0/dsv-darwin-x64"
                                - }
                                -})
                                +(*version.latestInfo)(<nil>)

                Test:           TestReadCache/Correct_case
2022/12/07 18:17:26 Wrong date format: 'wrong_date_content'
2022/12/07 18:17:26 Wrong file content: 'Wrong JSON string'
2022/12/07 18:17:26 Cached content too old
    --- FAIL: TestReadCache/Correct_case (0.01s)

=== FAIL: version TestReadCache (0.00s)

DONE 672 tests, 1 skipped, 4 failures in 11.527s

Merging. I verified the vulcheck results aren't really applicable as this is a cli, not server.
The lint issues are being bypassed as it's code that was already existing and got flagged for refactoring because of me adding some path updates to it.
Everything else looks good and was already approved. Will merge an updated changelog after this commits to kick off release.


Corrected couple of notebook links

Corrected notebook links for tensorflow and pytorch.

@thisisbhavin nice catch, thanks!


Support for OpenAI Whisper transcription from CLI

I have some Whisper JAX (https://huggingface.co/spaces/sanchit-gandhi/whisper-jax) transcription in .txt files and I'd like to use the CLI tool to convert them to .srt . Could that be added ?

I thought I included a sample, my mistake. Thank you.
[01:14.500 -> 01:15.500]  Okay.
[01:15.500 -> 01:17.500]  You know you can't smoke weed and drive, right?
[01:17.500 -> 01:19.500]  That's a DWI, man.
[01:19.500 -> 01:20.500]  Really?
[01:20.500 -> 01:21.500]  Yeah.
[01:22.500 -> 01:25.000]  And the reason I'm saying that is I can smell it, right?
[01:25.000 -> 01:29.000]  And I don't know if it's literally all over you.
[01:29.000 -> 01:32.000]  Or what's all that all over your shirt and your pants?
[01:32.000 -> 01:34.000]  Yeah, I know what I should say.
[01:34.000 -> 01:36.000]  It's like it's all over you, man.
[01:36.000 -> 01:38.000]  Hey, come on, dude.
[01:38.000 -> 01:40.000]  Yeah.
[01:40.000 -> 01:44.000]  So, if you drink and drive, it's a DWI, right?
[01:44.000 -> 01:47.880]  If you smoke weed and drive drive it's the DWI. They're both legal

Implemented in 1.8.0.


Update bundled LeanTween version

as discussed on forum here
The alternate is we hide leantween 2.3 inside the fungus namespace, so they don't conflict or interact. I'd prefer that if users know they have the latest leantween already there are working if they have the latest fungus

in OpenSource projects it's not best practice to embed external libs under own namespace.
much better to leave them as they are (maybe with a public dependencies doc)
it wouldn't harm casual developers using just fungus
and anybody having already a LeanTween lib in their project can easily solve the conflict by using the most recent version
so i suggest to take LeanTween out of the Fungus namespace!