Minimal example of Excel serialisation and de-serialisation

@jia-chenhua needs a minimal example of:

Loading an Excel file to a python object via pycel
Saving the Excel file to a file (serialisation)
Loading the file to a python object (de-serialisation)

This minimal example will help to address the feasibility of obfuscating the original Excel sheet, which is discussed here https://github.com/rangl-labs/netzerotc/issues/31

The working example is here: https://github.com/rangl-labs/netzerotc/tree/175e4566cdbf352decb50b16fbc17cf11e218774/environment/reference_environment_direct_deployment/hack

@moriartyjm and @masonlr , great news: I've finally resolved it, by looping through all cells involved in env.py to evaluate them all, and then serializing it to hard drive (using https://github.com/rangl-labs/netzerotc/blob/direct_action_on_deployment/environment/compiled_workbook_objects/evaluate_all_and_serialize.py), without the error message I mentioned above (when serializing after running test_reference_environment_direct_deployment.py), except for the following division-by-0 warnings due to empty cells in 'BREEZE' and 'GALE' from 2031 to 2050 (which we intentionally removed from the simplified workbook, and it should be fine):
Pathways2Net0.to_file("PathwaysToNetZero_Simplified_Compiled")
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P36") * (_C_("BREEZE!T36") / _C_("BREEZE!R36"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P37") * (_C_("BREEZE!T37") / _C_("BREEZE!R37"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P38") * (_C_("BREEZE!T38") / _C_("BREEZE!R38"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P39") * (_C_("BREEZE!T39") / _C_("BREEZE!R39"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P40") * (_C_("BREEZE!T40") / _C_("BREEZE!R40"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P41") * (_C_("BREEZE!T41") / _C_("BREEZE!R41"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P42") * (_C_("BREEZE!T42") / _C_("BREEZE!R42"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P43") * (_C_("BREEZE!T43") / _C_("BREEZE!R43"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P44") * (_C_("BREEZE!T44") / _C_("BREEZE!R44"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P45") * (_C_("BREEZE!T45") / _C_("BREEZE!R45"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P46") * (_C_("BREEZE!T46") / _C_("BREEZE!R46"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P47") * (_C_("BREEZE!T47") / _C_("BREEZE!R47"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P48") * (_C_("BREEZE!T48") / _C_("BREEZE!R48"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P49") * (_C_("BREEZE!T49") / _C_("BREEZE!R49"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P50") * (_C_("BREEZE!T50") / _C_("BREEZE!R50"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P51") * (_C_("BREEZE!T51") / _C_("BREEZE!R51"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P52") * (_C_("BREEZE!T52") / _C_("BREEZE!R52"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P53") * (_C_("BREEZE!T53") / _C_("BREEZE!R53"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P54") * (_C_("BREEZE!T54") / _C_("BREEZE!R54"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("BREEZE!P55") * (_C_("BREEZE!T55") / _C_("BREEZE!R55"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P36") * (_C_("GALE!T36") / _C_("GALE!R36"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P37") * (_C_("GALE!T37") / _C_("GALE!R37"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P38") * (_C_("GALE!T38") / _C_("GALE!R38"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P39") * (_C_("GALE!T39") / _C_("GALE!R39"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P40") * (_C_("GALE!T40") / _C_("GALE!R40"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P41") * (_C_("GALE!T41") / _C_("GALE!R41"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P42") * (_C_("GALE!T42") / _C_("GALE!R42"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P43") * (_C_("GALE!T43") / _C_("GALE!R43"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P44") * (_C_("GALE!T44") / _C_("GALE!R44"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P45") * (_C_("GALE!T45") / _C_("GALE!R45"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P46") * (_C_("GALE!T46") / _C_("GALE!R46"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P47") * (_C_("GALE!T47") / _C_("GALE!R47"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P48") * (_C_("GALE!T48") / _C_("GALE!R48"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P49") * (_C_("GALE!T49") / _C_("GALE!R49"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P50") * (_C_("GALE!T50") / _C_("GALE!R50"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P51") * (_C_("GALE!T51") / _C_("GALE!R51"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P52") * (_C_("GALE!T52") / _C_("GALE!R52"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P53") * (_C_("GALE!T53") / _C_("GALE!R53"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: division by zero
Eval: _C_("GALE!P54") * (_C_("GALE!T54") / _C_("GALE!R54"))
Values: 0 Div 0
Traceback (most recent call last):
  File "\WinPython64-<IP_ADDRESS>\python-3.7.7.amd64\lib\site-packages\pycel\excelutil.py", line 1269, in fixup
    return PYTHON_AST_OPERATORS[op](left_op, right_op)
ZeroDivisionError: float division by zero
Eval: _C_("GALE!P55") * (_C_("GALE!T55") / _C_("GALE!R55"))
Values: -380.262 Div 0

Despite the above warnings, the .pkl and .yml files are successfully serialized to hard drive, where the .yml file contains the values and formulae involved during looping through all cells involved in env.py. So these warnings above should be fine, since all formulae are stored in the serialized files, and once we set values/apply actions to fill the empty cells, it should get the correct values after .evaluate(). And this also passed the fixed policy test by running test_reference_environment_direct_deployment.py up to https://github.com/rangl-labs/netzerotc/blob/17aa28bd11c0c5525826c4426dca83e50025f572/environment/test_reference_environment_direct_deployment.py#L46, which now reads the extracted total capex, opex, revenue, carbon tax from the fixed-policy-filled original workbook (original xlsx, not the simplified xlsx, so the capex could depend on future, and hence the total capex doesn't match), and the differences between Excel workbook extracted values and env.py's output - for total opex, revenue, carbon tax - are in the order of 10^(-3) ~ 10^(-4), which I hope is acceptable.
Now, regarding the serialized .yml and .pkl: the .pkl is about ~2.4MB, containing non-readable text, which I think should be safe to push to GitHub; the .yml is about 236KB, containing information like the following:
cycles: false
excel_hash: 187d6d716602d4b366cc5b7d510ff9c1
cell_map:
  BREEZE!B25: 2020
  BREEZE!B26: 2021
...
  BREEZE!C24: Nuclear
  BREEZE!C25: =_C_("UK 2019!D7")
  BREEZE!C26: =_C_("BREEZE!C25") + ((_C_("BREEZE!C55") - _C_("BREEZE!C25")) / count(_R_("BREEZE!B25:B54")))
...
  GALE!P24: Capacity GW
  GALE!P25: 11.15
  GALE!P26: 12.53
...
  GALE!P35: 35.96
  GALE!P36:
  GALE!P37:
...
  GALE!AB24: GW green hydrogen
  GALE!AB25: =_C_("GALE!P25") * (_C_("GALE!T25") / _C_("GALE!R25"))
  GALE!AB26: =_C_("GALE!P26") * (_C_("GALE!T26") / _C_("GALE!R26"))
...
  GALE!AB36: =_C_("GALE!P36") * (_C_("GALE!T36") / _C_("GALE!R36"))
...
  Outputs!P24: =sum_(_R_("Outputs!P185:P186"), _R_("Outputs!P190:P191"))
  Outputs!P25: =_C_("Outputs!P187") + _C_("Outputs!P192")
  Outputs!P26: =_C_("Outputs!P236")
...
  Outputs!AI237: =_C_("Outputs!AI229") * _C_("Outputs!AI93")
...
  UK 2019!G57: 1.75641112725255
  UK 2019!G58: =sum_(_R_("UK 2019!G47:G57"))
filename: Pathways to Net Zero - Simplified.xlsx

So essentially, this .yml contains all cells involved in env.py with the column & row indices and either raw values or formulae. As you can see, some cells are empty because we removed them from the simplified xlsx file, and some formulae contain division and so it will raise a division-by-0 warning if an empty cell becomes the denominator. Since this .yml contains the full human-readable information of cells involved in env.py and their relations in terms of formulae, @moriartyjm , please let me know if I should push this .yml together with the .pkl to GitHub. Thanks!

Note: speed information when loading/de-serializing the .yml and .pkl files using https://github.com/rangl-labs/netzerotc/blob/17aa28bd11c0c5525826c4426dca83e50025f572/environment/compiled_workbook_objects/evaluate_all_and_serialize.py#L31:
start = time.time()
Pathways2Net0_Loaded = ExcelCompiler.from_file("PathwaysToNetZero_Simplified_Compiled")
end = time.time()
print(f"INFO: took {end - start} seconds to load from serialised file")
INFO: took 0.21702003479003906 seconds to load from serialised file
On average it could take 0.2 ~ 0.25 second to load. @moriartyjm , please let me know if this is still acceptable for env.reset purpose.

Nice work!

On average it could take 0.2 ~ 0.25 second to load. @moriartyjm , please let me know if this is still acceptable for env.reset purpose, which now invokes the Parameters() to initialize the env.param.

There might be workarounds for this. When we first create the env object we can load and pay the 0.2 second penalty. The env.reset() can mutate the pre-loaded pycel state – which I think will be an order of magnitude faster, but we need to test.

One tip: when linking to files like https://github.com/rangl-labs/netzerotc/blob/direct_action_on_deployment/environment/compiled_workbook_objects/evaluate_all_and_serialize.py it's handy to make a more permanent link.
You can do this by pushing "y" in GitHub when you visit the link.
For example, the link above turns into https://github.com/rangl-labs/netzerotc/blob/17aa28bd11c0c5525826c4426dca83e50025f572/environment/compiled_workbook_objects/evaluate_all_and_serialize.py
The general problem this addresses is that the first type of link points to the latest commit on a branch. If we update the branch then the link might end up pointing to updated content – this makes it confusing to revisit comment threads in the future when troubleshooting.


bugfix: add rabc setting for yurthub

What type of PR is this?

Uncomment only one  /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

/kind bug
What this PR does / why we need it:
yurthub need to list/watch nodepool resource, so we need add these RBAC rights to system:nodes group because a node certificate have been applied by yurthub.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

/assign @Fei-Guo
/assign @DrmagicE

Should you also change openyurt/hack/lib/build.sh file as well to replace `pki_path' ?

Should you also change openyurt/hack/lib/build.sh file as well to replace `pki_path' ?

@Fei-Guo build.sh has been updated, please check it.

/lgtm
/approve


Calling only -> no alignment

Hi,
I wanted to use your package, but ran into the issue that I do not want to realign my data - would it be possible to integrate a shortcut to skip alignment as well?
or is their something spacial done within the alignment, on which the rest is building?
If so, I had the impression that my multi-lane fastq was not accepted properly.
Could you set something up with an addition of *L{X}*R{1,2}.fastq.gz?
Cheers!

Hey @dansteiert,
The pipeline currently accepts FastQ files as input. This is to ensure the data has been properly aligned/recalibrated against the supported reference genome. This also acts as a safeguard to ensure it is compatible with any downstream steps.
With that being said, trimming and alignment will add some extra processing time but it's worth it. We cannot guarantee the accuracy of the results if you start with your own bam files.
If your current fastq files are split by lane (due to multiplexing across multiple flowcell lanes), please merge them prior to running the pipeline.
I hope this helps.
Best regards,
@skchronicles

Hi @skchronicles,
thank you for your input.
I understand the notion to keep it all within a single workflow, such that their is no confunding factors within the results.
Nevertheless I think it might be usefull for some users to proceed with their own bam file creation.
And basically request them to do things like a GATK best practises workflow type preprocessing.
With new and old technologies their might be always things needed to be performed, e.g. incorporating UMI tagged reads.
Just something to consider ;)
Cheers!


use all_gather_into_tensor instead of all_gather

when using allgather, the output is a list, and in the implementation of torch, the list will be flattened and unflattened, which will result in additional allocation of GPU memory and D2D operations. But these all gather operations already have a flat GPU memory, using all_gather_into_tensor  replaces all_gather will save GPU memory allocation and additional D2D operations.
additionally, batching all gatherers does not reduce the peak usage of GPU memory, so allgather_bucket_size has no effect.

@taozhiwei please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]

Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree [company="enflame-tech"]

r

@microsoft-github-policy-service agree company="enflame-tech"

@taozhiwei thx for the PR of changing all_gather list to tensor, which definitely can remove some overhead of unflatten and flatten.
One big concern I have is data misalignment. In this flat tensor solution, we assume tensor is evenly distributed, which might not be true in real world case. E.g. tensor size of 5 split over 2 GPUs. Although we can allocate flat buffer size as max_tensor_size_per_GPU * num_GPUs, we may still need to retain the information on which part of the tensor slice is padding and which part is real data.

allgather requires the input size must be the same size.if the tensor is not evenly partitioned

@taozhiwei thx for the PR of changing all_gather list to tensor, which definitely can remove some overhead of unflatten and flatten.
One big concern I have is data misalignment. In this flat tensor solution, we assume tensor is evenly distributed, which might not be true in real world case. E.g. tensor size of 5 split over 2 GPUs. Although we can allocate flat buffer size as max_tensor_size_per_GPU * num_GPUs, we may still need to retain the information on which part of the tensor slice is padding and which part is real data.

allgather requires the input size must be the same size. the place I modified was that using allgather would work properly, which proves that it can be evenly divided.

Hi @taozhiwei ,
Thx for the reply. I am aware that nccl right now does not support all-gatherV, but our current PartitionTensor() do support uneven split cases as residual.
Thus I suggest adding a top level flag/config as bool flat_all_gather_tensor to enable your implementation while keep our original version functional.

@taozhiwei thx for the PR of changing all_gather list to tensor, which definitely can remove some overhead of unflatten and flatten.
One big concern I have is data misalignment. In this flat tensor solution, we assume tensor is evenly distributed, which might not be true in real world case. E.g. tensor size of 5 split over 2 GPUs. Although we can allocate flat buffer size as max_tensor_size_per_GPU * num_GPUs, we may still need to retain the information on which part of the tensor slice is padding and which part is real data.

allgather requires the input size must be the same size. the place I modified was that using allgather would work properly, which proves that it can be evenly divided.

Hi @taozhiwei ,
Thx for the reply. I am aware that nccl right now does not support all-gatherV, but our current PartitionTensor() do support uneven split cases as residual.
Thus I suggest adding a top level flag/config as bool flat_all_gather_tensor to enable your implementation while keep our original version functional.

although partition_uniform support uneven split cases as residual,but full  call allgather,allgather requires that the input length must be the same size.so uneven split will cause a error in full . test_partition.py only tested the case of uniform segmentation.this error has not been reported, it should be that we have not encountered parameters that uneven split cases yet

@tjruwase @GuanhuaWang  please help review if it is possible to implement PartitionedTensor like this now?

@taozhiwei thx for the PR of changing all_gather list to tensor, which definitely can remove some overhead of unflatten and flatten.
One big concern I have is data misalignment. In this flat tensor solution, we assume tensor is evenly distributed, which might not be true in real world case. E.g. tensor size of 5 split over 2 GPUs. Although we can allocate flat buffer size as max_tensor_size_per_GPU * num_GPUs, we may still need to retain the information on which part of the tensor slice is padding and which part is real data.

allgather requires the input size must be the same size. the place I modified was that using allgather would work properly, which proves that it can be evenly divided.

Hi @taozhiwei ,
Thx for the reply. I am aware that nccl right now does not support all-gatherV, but our current PartitionTensor() do support uneven split cases as residual.
Thus I suggest adding a top level flag/config as bool flat_all_gather_tensor to enable your implementation while keep our original version functional.

although partition_uniform support uneven split cases as residual,but full  will call allgather,allgather requires that the input length must be the same size.so uneven split will cause a error in full funtion . test_partition.py only tested the case of even split.this error has not been reported, it should be that we have not encountered parameters that uneven split cases yet

hi @taozhiwei ,

it should be that we have not encountered parameters that uneven split cases yet

This is unsafe to say so. We do see uneven split on parameters (similar things also been observed by pytorch folks and used as the only example in their recent FSDP paper (as GPU 15 setting in Figure 3)

@taozhiwei thx for the PR of changing all_gather list to tensor, which definitely can remove some overhead of unflatten and flatten.
One big concern I have is data misalignment. In this flat tensor solution, we assume tensor is evenly distributed, which might not be true in real world case. E.g. tensor size of 5 split over 2 GPUs. Although we can allocate flat buffer size as max_tensor_size_per_GPU * num_GPUs, we may still need to retain the information on which part of the tensor slice is padding and which part is real data.

allgather requires the input size must be the same size. the place I modified was that using allgather would work properly, which proves that it can be evenly divided.

Hi @taozhiwei ,
Thx for the reply. I am aware that nccl right now does not support all-gatherV, but our current PartitionTensor() do support uneven split cases as residual.
Thus I suggest adding a top level flag/config as bool flat_all_gather_tensor to enable your implementation while keep our original version functional.

although partition_uniform support uneven split cases as residual,but full  will call allgather,allgather requires that the input length must be the same size.so uneven split will cause a error in full funtion . test_partition.py only tested the case of even split.this error has not been reported, it should be that we have not encountered parameters that uneven split cases yet

hi @taozhiwei ,

it should be that we have not encountered parameters that uneven split cases yet

This is unsafe to say so. We do see uneven split on parameters (similar things also been observed by pytorch folks and used as the only example in their recent FSDP paper (as GPU 15 setting in Figure 3)

If tensor can evenly divide, I will use all_gather_into_tensor , otherwise I will use broadcast, please help review again.

@taozhiwei thx for the PR of changing all_gather list to tensor, which definitely can remove some overhead of unflatten and flatten.
One big concern I have is data misalignment. In this flat tensor solution, we assume tensor is evenly distributed, which might not be true in real world case. E.g. tensor size of 5 split over 2 GPUs. Although we can allocate flat buffer size as max_tensor_size_per_GPU * num_GPUs, we may still need to retain the information on which part of the tensor slice is padding and which part is real data.
If tensor can evenly divide, I will use all_gather_into_tensor , otherwise I will use broadcast, please help review again.

@taozhiwei thx for the PR of changing all_gather list to tensor, which definitely can remove some overhead of unflatten and flatten.
One big concern I have is data misalignment. In this flat tensor solution, we assume tensor is evenly distributed, which might not be true in real world case. E.g. tensor size of 5 split over 2 GPUs. Although we can allocate flat buffer size as max_tensor_size_per_GPU * num_GPUs, we may still need to retain the information on which part of the tensor slice is padding and which part is real data.

allgather requires the input size must be the same size. the place I modified was that using allgather would work properly, which proves that it can be evenly divided.

Hi @taozhiwei ,
Thx for the reply. I am aware that nccl right now does not support all-gatherV, but our current PartitionTensor() do support uneven split cases as residual.
Thus I suggest adding a top level flag/config as bool flat_all_gather_tensor to enable your implementation while keep our original version functional.

although partition_uniform support uneven split cases as residual,but full  will call allgather,allgather requires that the input length must be the same size.so uneven split will cause a error in full funtion . test_partition.py only tested the case of even split.this error has not been reported, it should be that we have not encountered parameters that uneven split cases yet

hi @taozhiwei ,

it should be that we have not encountered parameters that uneven split cases yet

This is unsafe to say so. We do see uneven split on parameters (similar things also been observed by pytorch folks and used as the only example in their recent FSDP paper (as GPU 15 setting in Figure 3)

@GuanhuaWang  If tensor can evenly divide, I will use all_gather_into_tensor , otherwise I will use broadcast, please help review again.

@taozhiwei thx for the PR of changing all_gather list to tensor, which definitely can remove some overhead of unflatten and flatten.
One big concern I have is data misalignment. In this flat tensor solution, we assume tensor is evenly distributed, which might not be true in real world case. E.g. tensor size of 5 split over 2 GPUs. Although we can allocate flat buffer size as max_tensor_size_per_GPU * num_GPUs, we may still need to retain the information on which part of the tensor slice is padding and which part is real data.

@GuanhuaWang If tensor can evenly divide, I will use all_gather_into_tensor , otherwise I will use broadcast, please help review again.

@GuanhuaWang @tjruwase  If tensor can evenly divide, I will use all_gather_into_tensor , otherwise I will use broadcast, please help review again.

This is unsafe to say so. We do see uneven split on parameters (similar things also been observed by pytorch folks and used as the only example in their recent FSDP paper (as GPU 15 setting in Figure 3)

@GuanhuaWang If tensor can evenly divide, I will use all_gather_into_tensor , otherwise I will use broadcast, please help review again.

@GuanhuaWang @tjruwase If tensor can evenly divide, I will use all_gather_into_tensor , otherwise I will use broadcast, please help review again.

@GuanhuaWang @tjruwase Could you please help me review again? The code has been modified for a long time。

use https://github.com/microsoft/DeepSpeed/pull/4705 replace


[V4-beta.7] ion-datetime not bind as ISO-8601 format

Bug Report
Ionic Info
Ionic:

   ionic (Ionic CLI)          : 4.1.1 (/Users/joeldaros/.nvm/versions/node/v8.11.3/lib/node_modules/ionic)
   Ionic Framework            : @ionic/angular 4.0.0-beta.7
   @angular-devkit/core       : 0.7.5
   @angular-devkit/schematics : 0.7.5
   @angular/cli               : 6.1.5
   @ionic/ng-toolkit          : 1.0.7
   @ionic/schematics-angular  : 1.0.5

Cordova:

   cordova (Cordova CLI) : 8.0.0
   Cordova Platforms     : ios 4.5.5
   Cordova Plugins       : cordova-plugin-ionic-keyboard 2.1.2, cordova-plugin-ionic-webview 2.0.2, (and 7 other plugins)

System:

   ios-deploy : 2.0.0
   NodeJS     : v8.11.3 (/Users/joeldaros/.nvm/versions/node/v8.11.3/bin/node)
   npm        : 6.4.0
   OS         : macOS High Sierra
   Xcode      : Xcode 9.4 Build version 9F1027a

Describe the Bug
The ion-datetime component do not bind his value to a ISO-8601 format when changed.  It binds the date to a object:
{
  "dateBk": {
    "day": { "text": "21", "value": 21, "columnIndex": 0 },
    "month": { "text": "Aug", "value": 8, "columnIndex": 1 },
    "year": { "text": "2017", "value": 2017, "columnIndex": 2 }
  }
}

Example:
The initial date (ISO-8601) of component is correct:

When you pick another date from the component it binds to a object instead of a string in 8601 format like described in documentation:

Steps to Reproduce
Steps to reproduce the behavior:

Change the date from the ion-datetime component
Check the results.

Related Code
https://github.com/MrSparklle/Inc4

At present, my solution is
private _startDate: string;

get startDate() {
  return this._startDate;
}

set startDate(value: any) {
  if (typeof (value) !== 'string') {
    const { year, month, day, hour } = value,
      date = new Date(year.value, month.value - 1, day.value, hour.value);
    this._startDate = format(date, 'YYYY-MM-DDTHH:00:00');
  } else {
    this._startDate = value;
  }
 }

Also just hit this issue, docs all say that it reads and returns ISO format datetime as string.  As it did in V3.
It should continue to do so or the documentation should be updated and this added as a breaking change.

I agree with @ghenry22
Especially since it's recommended using the date-fns library to work with dates. I can't find a simple way to create a new object with date-fns only using hours and minutes (since I use the datetime picker only for the time) to calculate a duration (or difference in minutes). Except (again) manually creating it.

Also hit this issue, and I find a another bug that it can't select year with 2018

@woodstream The 2018 year bug has been solved in beta-7, just upgrade your @ionic/angular

@brandyscarney Please, promote this to Backlog for correction.

Any idea on when this will be fixed?

@kensodemann would be great to get this fixed, I am working on a project that uses the dateTime picker and I have been working around this on other things rather than building in work arounds left and right.

What I have done in one of my own projects until I can get this fixed is to not do the two-way binding on ngModel but to do the property binding on value and the event binding on ionChange (which is all the ngModel binding really does anyhow), and then if the evt.target.value is a string I use it, if it is an object I pull it apart to create a string.
Then I can easily swap that out once I get a fix in and it is released.

This issue has been fixed in https://github.com/ionic-team/ionic/pull/15833 and https://github.com/ionic-team/ionic/pull/15907 released in (Beta-13).


window size should not be higher than screen size

My laptop has a 1366x768 screen resolution. I am not sure atm how sciencefair sets the window size, but at least on Ubuntu 16.04 when running Unity as Desktop environment the window is drawn with such a height that the lower part is cut off. It is probably just a few pixels that are cut off, nothing important, but it is hard to actually know that for sure/find out.

@step21 good point! Currently the app window is created with the following settings:
      height: 750,
      width: 1080,
      minHeight: 750,
      minWidth: 1080,

The reason for having a restriction in the first place is that the interface doesn't work with a screen much small than this, but given 768 is probably quite a common screen resolution height we should lower the height default and minimum.


/docs/user-guide/quick-start/ contains incorrect expose instructions

The quick start guide begins with the creation of a deployment/service, 'my-nginx':
kubectl run my-nginx --image=nginx --replicas=2 --port=80
The next step is to expose the newly created service with the following command:
kubectl expose rc my-nginx --target-port=80 --type=LoadBalancer
However, when executed against the default cluster built with kube-up.sh on AWS, this results in:
Error from server: replicationcontrollers "my-nginx" not found
It would appear that the proper step in this case is actually to expose the deployment:
$ kubectl expose deployment my-nginx --target-port=80 --type=LoadBalancer
service "my-nginx" exposed
If this is the correct step to take, the documentation should be updated to reflect it.

This looks to have been resolved.


⚠ Bofrost.lu has degraded performance

In bf56737, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1149 ms

Resolved: Bofrost.lu performance has improved in 86cfbfc after 13 minutes.


Curl -k equivalent

Is there any way to do a curl -k equivalent with the bridge devices to provide an insecure SSL connection?  I am trying to run commands on a Vizio TV which work using the command line and curl -k from this API: https://github.com/exiva/Vizio_SmartCast_API
If I don't use the -k I get: "Error on calling url to change device state: https://192.168..."

So looking into this a bit more. Could you post the error from the program output log that occurs from the https call?

Also, an alternative is to use the command type for ha-bridge and call curl itself.

included in the community information https://github.com/bwssytems/ha-bridge/wiki/Informational-Items-Collected-from-the-Community#q-curl--k-equivalent


Introduced fromJust

The fromJust function extracts the element out of a Just and throws an
error if its argument is Nothing.

Sorry for not responding to this PR, I missed it when it came in.
What do you think about the unsafe function in https://github.com/circuithub/elm-maybe-extra/pull/5?

It's a complete copy of what I did in the patch. I think if you accept PR #5, you could close this one without merging.
One small remark: imho fromJust would be a more descriptive name for the function. :)

Since #5 was merged, closing this as suggested. Thanks!

So, FYI unsafe was removed from https://github.com/circuithub/elm-maybe-extra/pull/5, but it is available in http://package.elm-lang.org/packages/sindikat/elm-maybe-experimental/1.0.0.
I feel a bit bad for not including it since there is a proposal for it at https://github.com/elm-lang/core/issues/215, but you can always use
  x = mx ? Debug.crash "Invariant violated: mx should have been checked by f"

which I feel might be better since you're probably going to document the broken invariant with something like this anyway:
  # Invariant: mx is checked by f earlier on
  x = Maybe.unsafe mx


Instances of "play audio" node in separate flows use same voice though voice selection differs

The play audio node behaves as if the voice selection is global to the node-red instance, i.e. when I have multiple play audio nodes that use different voices, it uses the voice I last selected and deployed in all play audio nodes.
What are the steps to reproduce?
I created two flows, each of which has a play audio node. I set the voices on each play audio node to be different. For example, in one flow it's Samantha, while the other uses Fred.
What happens?
When I deploy, the voice heard on both flows will be the same voice, e.g. only Fred. When I look at the settings of both nodes, the voices are correct, i.e. they are the ones I originally selected, and they are different voices.
What do you expect to happen?
My assumption was that the voice selected would be used in each play audio node.
Please tell us about your environment:
Node-RED version: 0.18.2
node.js version:  8.9.4
npm version: 5.6.0
Platform/OS: Raspberry Pi / linux 4.9.35-v7+ arm LE
Browser: chrome on Mac OS 10.13.3

Which node exactly are you using? From what module?
If it's from a node-red-contrib- module, then you should raise an issue against their repository. We don't have any play audio nodes in the Node-RED project.

It's called node-red-contrib-play-audio. I assumed it was part of the Node-RED project because I never explicitly installed it. I'll go to their project. Thanks!


[Constraint System] Diagnose conversion to non-optional argument mismatch

The Constraint System fails to diagnose failures and suggest a property wrapper fix for optional expressions/optional payloads. Suggesting a fix to remove the optional chaining is a good approach since doing so allows the constraint system to properly diagnose these edge cases.
rdar://98019501

@swift-ci please smoke test

Note about description - let's use "solver" instead of "system" because system is really just the information side.


Adding Pagintion for workflows and agents page

Summary
This PR fixes #583

[x] My code follows the code style of this project and passes make format.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] My change has adequate unit test coverage.

The changes in this pull request include:

Updating the list functions in agents.py and workflows.py to include pagination functionality.
Adding a total_pages field to the response models AgentList and WorkflowList in response.py.
Adding a new component DataTablePagination to handle pagination in the UI.
Updating the Agents and Workflows pages to include pagination functionality.

Overall, the changes look good and provide a way to paginate through the list of agents and workflows. The code is well-structured and follows the existing conventions. Nice work, @elisalimli! 👍
One suggestion I have is to add some error handling in case there is an exception while fetching the agents or workflows. It would be helpful to display an error message to the user if something goes wrong.

@RutamBhagat please resolve merge conflicts.

@RutamBhagat please resolve merge conflicts.

Um, I am sorry but I do not know how to do that


Update example in documentation

index.rst needs to be updated as follows after #2 is merged:

pyRiemann-qiskit: Qiskit wrapper for pyRiemann
=============================================================

.. raw:: html

       <a href="<link to example>">

api.rst needs to be updated as follows after #2 is merged:

.. _api_ref:

=============
API reference
=============

Classification
--------------
.. _classification_api:
.. currentmodule:: pyriemann_qiskit.classification

.. autosummary::
    :toctree: generated/
    :template: class.rst

    QuanticBase
    QuanticSVM
    QuanticVQC

Closing as this was finally implemented as part of #2:

in index.rst

in api.rst


Tracking issue for upstream order_tracking bug

xref: https://github.com/h5py/h5py/issues/1385
It does somewhat seem like it is a bug in hdf5. I wonder if we can recreate it in netcdf4 and get them involved too!

I've come up with a workaround which also has the nice advantage to be a bit more performant when creating millions of attributes ;-)
https://github.com/h5py/h5py/pull/2042
FYI: @paugier

I've tested that patch a bit. So the track_order problem is avoided then.
But a major pain point is, that every change of an attribute would delete and recreate it. With track_order=True this would move the changed attribute to the back of the list.
I think we need to fix this either in h5py (allowing to modify existing attributes with same dtype) or in h5netcdf. It seems, that netCDF4 keeps this order regardless of an attribute has changed value and/or type.

Does netCDF4 suffer from the same issue? or no because they don't rename variable like h5py does?

Yes, netCDF4 suffers from the same issue. At least first tests suggest this.

import netCDF4
with netCDF4.Dataset("test.nc", mode="w") as nc:
    g = nc.createGroup("grp0")
    g.test12 = 12
    g.test0 = 0
    g.test1 = 1
    g.test2 = 2
    g.test3 = 3
    g.test4 = 4
    g.test5 = 5
    g.test6 = 6
    g.test7 = 7
    g.test8 = 8
    g.test9 = 9
    g.test10 = 10
    g.test11 = 11

with netCDF4.Dataset("test.nc", mode="r") as nc:
    print(nc["grp0"])

with netCDF4.Dataset("test.nc", mode="a") as nc:
    g = nc["grp0"]
    g.test8 = 4
    g.test9 = "3"
    g.test1 = 42.

with netCDF4.Dataset("test.nc", mode="r") as nc:
    print(nc["grp0"])

<class 'netCDF4._netCDF4.Group'>
group /grp0:
    test12: 12
    test0: 0
    test1: 1
    test2: 2
    test3: 3
    test4: 4
    test5: 5
    test6: 6
    test7: 7
    test8: 8
    test9: 9
    test10: 10
    test11: 11
    dimensions(sizes):
    variables(dimensions):
    groups:
<class 'netCDF4._netCDF4.Group'>
group /grp0:
    test12: 12
    test0: 0
    test2: 2
    test3: 3
    test4: 4
    test5: 5
    test6: 6
    test7: 7
    test8: 4
    test10: 10
    test11: 11
    test1: 42.0
    test9: 3
    dimensions(sizes):
    variables(dimensions):
    groups:

@hmaarrfk So, only if the data type is the same, netCDF4 just modifies the entry. h5py recreates in any case atm. So we would need to fix either in h5py or h5netcdf.
But I'm wondering why test9 is listed after test1 although it has been changed the other way round.

But I'm wondering why test9 is listed after test1 although it has been changed the other way round.

Further tests indicate, that this might be connected with attr_phase_change setting.
It looks like attributes in compact storage are somehow handled differently from attributes in dense storage.

netcdf-c deletes the attribute in the hdf5 file (H5Adelete) when renaming. It keeps the renamed attribute in internal netcdf-c cache, so it get's written to the file in consequence. The data will be written most likely before the objects go out of scope or on file close/flush.
https://github.com/Unidata/netcdf-c/blob/e7e2f40188a35011feb25a13b4a129b495a28043/libhdf5/hdf5attr.c#L153-L245
Similar will be true, when overwriting an attribute. That's most likely the reason why netcdf-python/netcdf-c does not suffer from this weird rename/create bug as they actually do not rename anything.

@hmaarrfk with h5py 3.7.0 out our tracking test:
https://github.com/h5netcdf/h5netcdf/blob/9c0269fa78286dd5a3547a040dd3b6c8902e9a68/h5netcdf/tests/test_h5netcdf.py#L1597-L1614
isn't failing any more. We would need to reevaluate the situation.

see https://github.com/h5netcdf/h5netcdf/pull/173

Looks like we can reinstate track_feature=True as default.


Catalan translation updates 4 new in ui, 20 new in web access

Catalan translation update (4 new in ui)
Catalane translation update (20 new in web access)

Thanks Santiago :+1:


tap_auditor: check formulae names in autobump.txt

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

We audit all other formulae lists and this allows identifying when a removed formulae has leftover/outdated references.
In future, may want to refactor autobump handling to give it its own DSL so that we can explicitly label certain formulae as non-autobumped for a specific reason (e.g. upstream autobumps, consistently broken for last bump attempts, etc.)

I'm now OK with a DSL now that we have brew bump --auto. It didn't make sense before when it was tied to a random homebrew-core workflow but if we unhide the --auto flag and support it for third-party taps then the DSL makes sense.

Alternative suggestion to a DSL: https://github.com/Homebrew/brew/pull/18227#discussion_r1741071946

Probably can discuss further either in dedicated issue or whenever anyone gets chance for PR.


Easy way to integrate Commoner with Webpack

This could be a webpack loader, or as simple as some guidance in the Readme.

@mdebbar for webpack you can just use resolve.alias without any loader or preprocessing


Fix minor neo-cli compile errors

Didn't notice #388 makes these fixes. Abandoning this PR

Minds are connected...whsushah


[Bug]: Extension Methods only work directly on Builder

Testcontainers version
3.9.0
Using the latest Testcontainers version?
 (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.10.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 55
  Running: 22
  Paused: 0
  Stopped: 33
 Images: 1018
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: <IP_ADDRESS>-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.2GiB
 Name: docker-desktop
 ID: da77bd29-9b14-4aeb-87d1-188e94664cfa
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  <IP_ADDRESS>/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

What happened?
If the the extension methods WithNetworkAliases() or WithNetwork() are not called directly on the builder class they do not have any effect.
In the example below the container should be created with a connection to the network.
But if the methods are executed on the builder variable later they do not have any effect.
Relevant log output
-

Additional information
Example for reproduction:
// See https://aka.ms/new-console-template for more information

using DotNet.Testcontainers.Builders;
using Testcontainers.MsSql;

var network = new NetworkBuilder()
    .WithName(Guid.NewGuid().ToString("D"))
    .Build();

await network.CreateAsync();

var builder = new MsSqlBuilder();
    // This would work
    // .WithNetwork(network)
    // .WithNetworkAliases("test-mssql")
    // .WithCleanUp(true);

// This does not work
builder
    .WithNetwork(network)
    .WithNetworkAliases("test-mssql")
    .WithCleanUp(true);

var container = builder.Build();
await container.StartAsync();
Console.WriteLine("Hello, World!");

This works as expected and is by design. We do not change the internal state of the builder, instead, we create and return a new instance. You need to assign the returned value to the builder variable again. This allows the reuse of configuration e.g. for A/B testing.

Ahh, alright. So this should work?
builder = builder
    .WithNetwork(network)
    .WithNetworkAliases("test-mssql")
    .WithCleanUp(true);


Add Exoscale host provider

Exoscale is a Swiss cloud provider and one of the provider currently
supported in Docker Machine. Its addition in Rancher UI follows a logic
similar to Amazon EC2 with a multi-step wizard:

Ask for credentials.
Grab the list of security groups.
Ask which security group to use and propose to create the default
one if it doesn't exist.
Create the default security group if requested.
Grab the instance profiles and available disk sizes.
Let the user choose a name, instance profile and disk size.

Currently, only one availability zone is available. Therefore, there is
no drop-down list to choose one.
While there are several images available, we enforce the use of Ubuntu
Trusty. It would be possible to let the user choose, but this needs some
work to know which images would work and which wouldn't.
The default ingress rules also include the Docker daemon port. Docker
Machine needs it for the first time installation.
This PR needs https://github.com/rancherio/cattle/pull/667 to work as expected since Exoscale API only works over HTTPS. Also, api.exoscale.ch needs to be whitelisted. This can be done by setting the CATTLE_API_PROXY_WHITELIST=api.exoscale.ch environment variable. I will also do a PR for cattle for that.
If you want to test, you can register with the DOCKERMACHINEROCKS coupon code to get 20CHF of credit (the code works several times).

@vincentbernat This looks good.. I'll take a closer look and merge once rancher/cattle#667 and rancher/cattle#670 are merged so I can test it.

@vincent99 I've merged the cattle PRs.  Should land in latest in the morning.

Merged in #228 with some tweaks for the Ember 1.13 upgrade merged after you started;  Thanks @vincentbernat!

Great! And thanks for fixing the two bugs that I didn't notice!


Tekton-pipelines project can't be deleted on ocp-3.11

Issue
Tekton-pipelines project can't be deleted on ocp-3.11
oc delete project tekton-pipelines --force=true --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "tekton-pipelines": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

Command executed
oc delete -f https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml
Error from server (Conflict): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": Operation cannot be fulfilled on namespaces "tekton-pipelines": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": podsecuritypolicies.policy "tekton-pipelines" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": clusterroles.rbac.authorization.k8s.io "tekton-pipelines-admin" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": serviceaccounts "tekton-pipelines-controller" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": clusterrolebindings.rbac.authorization.k8s.io "tekton-pipelines-controller-admin" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": customresourcedefinitions.apiextensions.k8s.io "clustertasks.tekton.dev" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": customresourcedefinitions.apiextensions.k8s.io "images.caching.internal.knative.dev" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": customresourcedefinitions.apiextensions.k8s.io "pipelines.tekton.dev" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": customresourcedefinitions.apiextensions.k8s.io "pipelineruns.tekton.dev" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": customresourcedefinitions.apiextensions.k8s.io "pipelineresources.tekton.dev" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": customresourcedefinitions.apiextensions.k8s.io "tasks.tekton.dev" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": customresourcedefinitions.apiextensions.k8s.io "taskruns.tekton.dev" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": services "tekton-pipelines-controller" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": services "tekton-pipelines-webhook" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": configmaps "config-artifact-bucket" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": configmaps "config-logging" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": deployments.apps "tekton-pipelines-controller" not found
Error from server (NotFound): error when deleting "https://storage.googleapis.com/tekton-releases/previous/v0.3.1/release.yaml": deployments.apps "tekton-pipelines-webhook" not found

Which tekton k8s resource is blocking to delete the namespace/project

This bash script saved my life as it removes the finalizer kubernetes and post the project updated
https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-497542245

IMO This more like a bug in k8/openshift than tekton

Yes, as @chmouel said :+1:. However, we could document this somewhere?

Document k8 gotchas in pipeline cli ? IMHO: People would probably google it and find this brilliant issue to get to Charle’s bash script link to fix it :)

Closing it as it doesn't seem to be a tekton bug :angel:
/close


Add actual date and time to 'timeago' dates (#5085)

This is a fix for the issue #5085
As per @dstufft every date time is stored in UTC so I added "UTC" in the end. Let me know if that's not what I was supposed to do.

Added a tooltip for date time to 'timeago' dates with timezone

I think the right thing to do here would be to localize the datetime prior to using it, and then add %Z to the formatting string:
>>> dt
datetime.datetime(2018, 12, 26, 13, 36, 5, 789013)
>>> import pytz
>>> pytz.utc.localize(dt)
datetime.datetime(2018, 12, 26, 13, 36, 5, 789013, tzinfo=<UTC>)
>>> _.strftime('%Y-%m-%d %H:%M %Z')
'2018-12-26 13:36 UTC'

Hi @di! Thanks for the review. I know printing "UTC" was plain stupid :P
I actually ran into a problem regarding the all_releases query object in /warehouse/packaging/views.py which passes the all_releases list to the template, as it can't be modified so I converted it to a dict but then 1 test failed and moreover, we can't import pytz in the template. Any help appreciated

I'd recommend adding it as a filter instead, so we could use it in the Jinja template whenever we need to:
{{ hversion.created.strftime('%Y-%m-%d %H:%M') | localize_datetime }}

Thanks, @di!
I've added a filter to localize datetime and its corresponding unit test.
Also, I found another filter to format datetime so, i removed the strftime() and used that instead.

Thanks @darkHarry!

It was Awesome. Thanks a lot, @di.


networking configuration on instance

Nomad's docker driver doesn't take into consideration when it is running in a Docker Desktop environment like it has to on Mac.
There are two ways to expose a container's port: using network_mode = "host", and using static port configuration. Nomad tries to be smart and skips setting up port forwarding when network mode is configured to host. This works fine with docker linux, but not with docker desktop on macos, because network_mode = "host" is not supported.
In our current configuration, registry publishes its port on <IP_ADDRESS>.:7979. Nginx tries to proxy registry.local.pipal.in to this address and fails.
I didn't notice this issue locally because I had changed the job template to use host.docker.internal:{{ .Port }} instead of the one in the codebase {{ .Address }}:{{ .Port }}, but I noticed it when trying to deploy rajdhani project to capstone.
I think a good workaround, and to avoid such inconsistencies in the future, would be to use the extra_hosts configuration to allows forward host.docker.internal to the host.
Or, I think Nomad also supports using templates in task configurations.
We could use the private IP with static port for preset jobs, and then not have to do any special workarounds.

I would like to avoid spending time on the quirks of running Nomad on Mac,
at least in these initial stages. Would it be okay for you if we figure out
a way to run Nomad on your dev server, but make it look like a local one
with some ssh port forwarding magic?
On Thu, Apr 20, 2023 at 7:10 PM Kaustubh Maske Patil <
@.***> wrote:

Nomad's docker driver doesn't take into consideration when it is running
in a Docker Desktop environment like it has to on Mac.
There are two ways to expose a container's port: using network_mode =
"host", and using static port configuration. Nomad tries to be smart and
skips setting up port forwarding when network mode is configured to host.
This works fine with docker linux, but not with docker desktop on macos,
because network_mode = "host" is not supported.
I didn't notice this issue locally because I had changed the job template
to use host.docker.internal:{{ .Port }} instead of the one in the
codebase {{ .Address }}:{{ .Port }}, but I noticed it when trying to
deploy rajdhani project to capstone.
I think a good workaround, and to avoid such inconsistencies in the
future, would be to use the extra_hosts
https://developer.hashicorp.com/nomad/docs/drivers/docker#extra_hosts
configuration to allows forward host.docker.internal to the host.
Or, I think Nomad also supports using templates in task configurations.
We could use the private IP with static port for preset jobs, and then not
have to do any special workarounds.
—
Reply to this email directly, view it on GitHub
https://github.com/pipalacademy/capstone/issues/54, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAB3ENKQH4GWNQHLCZG6CDXCE4DJANCNFSM6AAAAAAXFQXNOU
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

That would work, yes.
On Thu, Apr 20, 2023, 7:33 PM Anand Chitipothu @.***>
wrote:

I would like to avoid spending time on the quirks of running Nomad on Mac,
at least in these initial stages. Would it be okay for you if we figure out
a way to run Nomad on your dev server, but make it look like a local one
with some ssh port forwarding magic?
On Thu, Apr 20, 2023 at 7:10 PM Kaustubh Maske Patil <
@.***> wrote:

Nomad's docker driver doesn't take into consideration when it is running
in a Docker Desktop environment like it has to on Mac.
There are two ways to expose a container's port: using network_mode =
"host", and using static port configuration. Nomad tries to be smart and
skips setting up port forwarding when network mode is configured to host.
This works fine with docker linux, but not with docker desktop on macos,
because network_mode = "host" is not supported.
I didn't notice this issue locally because I had changed the job template
to use host.docker.internal:{{ .Port }} instead of the one in the
codebase {{ .Address }}:{{ .Port }}, but I noticed it when trying to
deploy rajdhani project to capstone.
I think a good workaround, and to avoid such inconsistencies in the
future, would be to use the extra_hosts
https://developer.hashicorp.com/nomad/docs/drivers/docker#extra_hosts
configuration to allows forward host.docker.internal to the host.
Or, I think Nomad also supports using templates in task configurations.
We could use the private IP with static port for preset jobs, and then
not
have to do any special workarounds.
—
Reply to this email directly, view it on GitHub
https://github.com/pipalacademy/capstone/issues/54, or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAAB3ENKQH4GWNQHLCZG6CDXCE4DJANCNFSM6AAAAAAXFQXNOU
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

—
Reply to this email directly, view it on GitHub
https://github.com/pipalacademy/capstone/issues/54#issuecomment-1516390538,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AI7MKYNWTQ7AC5Q47SA43CLXCE645ANCNFSM6AAAAAAXFQXNOU
.
You are receiving this because you authored the thread.Message ID:
@.***>


"Lăicăi" is listed in the drop-down menu

Description:
"Lăicăi" is listed in the drop-down menu alongside its corresponding county and commune, according to the law
Precondition:
The website is up an running.

Step 1
Type "Lăicăi" in the search bar.

Expected results
The location is listed in the drop-down menu as "Lăicăi, Argeș (Cetățeni)".


Fix benchmarking MobileNetV2 on S20 GPU in Linalg on tensor path.

See https://github.com/google/iree/pull/5282 for more details.
The Linalg on tensors path should work with -iree-spirv-enable-vectorization, also added the forgotten flag.

Triggered the test in https://buildkite.com/iree/iree-android-benchmark/builds/2227


jenkins_user experimental password setting is broken

I have been trying to get the jenkins_user experimental type to work with hudson.security.HudsonPrivateSecurityRealm under 1.641.  I haven't yet determined if is an existing bug due to the combination of resources or a recent changes in jenkins.  The cli jar returns java.io.EOFException errors as below until the "administrative" user's account is opened and then saved (without modification) in the WWW UI.
==> master: Error: Could not prefetch jenkins_num_executors provider 'cli': Execution of '/bin/java -jar /usr/lib/jenkins/jenkins-cli.jar -s http://localhost:8080 -i /usr/lib/jenkins/admin_private_key groovy /usr/lib/jenkins/puppet_helper.groovy get_num_executors' returned 255: java.io.EOFException
==> master: 	at java.io.DataInputStream.readBoolean(DataInputStream.java:244)
==> master: 	at hudson.cli.Connection.readBoolean(Connection.java:95)
==> master: 	at hudson.cli.CLI.authenticate(CLI.java:573)
==> master: 	at hudson.cli.CLI._main(CLI.java:484)
==> master: 	at hudson.cli.CLI.main(CLI.java:390)

I finally looked in the jenkins log and it appears that the password in not being set any longer.
Exception in thread "Thread-191" org.acegisecurity.userdetails.UsernameNotFoundException: Password is not set: admin

Opening the user in the WWW UI and saving it without any change sets the password to empty and the CLI jar starts working again.

This problem does not occur if you set the password to be a plain text string rather than the hash form (#jbcrypt:...).  This is a usable work around but it causes the type to be non-idempotent.

@jhoblitt I tried setting as suggested:
jenkins_user {'jadmin':
  password => 'changeme',
}

but still getting java.io.EOFException
[root@jenkins ~]# /bin/java -jar /usr/lib/jenkins/jenkins-cli.jar -s http://localhost:8080 -i /usr/lib/jenkins/admin_private_key groovy /usr/lib/jenkins/puppet_helper.groovy who_am_i
java.io.EOFException
        at java.io.DataInputStream.readBoolean(DataInputStream.java:244)
        at hudson.cli.Connection.readBoolean(Connection.java:95)
        at hudson.cli.CLI.authenticate(CLI.java:573)
        at hudson.cli.CLI._main(CLI.java:484)
        at hudson.cli.CLI.main(CLI.java:390)

What did you do to make this work around work?

@cdenneen There is no method in puppet_helper.groovy named who_am_i, so your example command will always fail.  A better test would be /bin/java -jar /usr/lib/jenkins/jenkins-cli.jar -s http://localhost:8080 -i /usr/lib/jenkins/admin_private_key who-am-i.  If that sanity check fails, your jenkins installation is broken.

[vagrant@jenkins ~]$ /bin/java -jar /usr/lib/jenkins/jenkins-cli.jar -s http://localhost:8080 -i /usr/lib/jenkins/admin_private_key who-am-i
java.io.EOFException
        at java.io.DataInputStream.readBoolean(DataInputStream.java:244)
        at hudson.cli.Connection.readBoolean(Connection.java:95)
        at hudson.cli.CLI.authenticate(CLI.java:573)
        at hudson.cli.CLI._main(CLI.java:484)
        at hudson.cli.CLI.main(CLI.java:390)

So this would appear as if it's broken but can't imagine how since UI works just fine. Is there a way to DEBUG why this error is occuring?

The master log sometimes has different information.  I suggest you try to get a working jenkins installation with your desired authentication configuration before trying to puppetize it.

I'm closing this since I assume it was resolved?


Rename current panel/layout 'stretch' behavior to CSS 'fit-content' and make 'stretch' match the equivalent WPF behavior

Currently alignment: stretch works more like https://developer.mozilla.org/en-US/docs/Web/CSS/fit-content. If a vertical list of items has horizontal alignment as 'stretch', each item will be sized to the maximum width of the items in the list. In WPF however, stretch means to fill the entire available width or height (and YUI 'stretch' was intended to match this behavior). The current behavior is very useful though, so it should be preserved under the name 'fit-content'.

Fixed in 6d4e799 in a different way -- the new behavior is that stretch alignment matches the fit-content when the associated panel size is "auto", and will fully stretch to fill the available space when the associated size is explicit or "stretch". This gives the best of both worlds without having to complicate the API with further options


(PDB-3470) Add pdbbox and use it in tests; test osx in travis

See the documentation/ and .travis.yml changes for further details.
Note that this adds additional inter-dependent utility scripts to
ext/bin, but we don't want to add ext/bin to the PATH because so far,
the command names in ext/bin haven't been chosen with that in
mind (i.e. sha256sum would shadow the Linux version), so we refer to
them with explicit paths (sometimes generated via basename).
In the longer run we might want to adjust the relevant command names,
or perhaps move "utility" commands to ext/util/, to make ext/bin
suitable for inclusion in the PATH.
Remove the ulimit -u settings because it removes another platform
variance (disallowed on travis osx hosts), and it looks like we may no
longer need it, but if we see the OutOfMemoryError exceptions
return (2080a332b40f95cdf564a65a6bf540704273b583), we can restore it.

Added the "don't merge" for now while we make sure the (upcoming) pdbext pr is set, etc.

Test PASSed

CLA signed by all contributors.

Test PASSed

Test PASSed

Test FAILed

Test FAILed

Test FAILed

Test FAILed

Test FAILed

Test FAILed

Test FAILed

Test FAILed

Test FAILed

Test PASSed


Respond to user feedback on data browse screen

Upon demoing data browse screen, multiple users noted that the page layout carried some usability issues. In particular, the large hero image pushes the content almost entirely off the bottom of the screen, so that users have to scroll before they can see it. As the hero is not particularly instructive, users are a bit confused on initially seeing the screen- they'd prefer the list be visible right when you land on the page. Users also noted the list entries themselves are rather tall. In general, they'd prefer to see more at once without as much scrolling.

New screen uploaded to Invision...


Add scaling to the curve editor's handles

This fixes #50617
Before the fix the curve editor's handles have no scaling applied on Hi-DPI displays

With the fix scaling is applied:

Cherry-picked for 3.4.

Cherry-picked for 3.3.3.


Pitfall-Atari2600 scenario is broken

Pitfall-Atari2600 seems to have a bad done check (when the timer runs out) and also does not start with the timer running (the player needs to hit something to start the timer).

Looks like the same issue exists on ALE: https://github.com/mgbellemare/Arcade-Learning-Environment/issues/268

Given that the Atari scenarios are straight ports of the ALE versions, there could be a lot of those. Might be good to skim through their issues to find ones that I might have overlooked.


There is a UI issue on the notification page

Bagisto Version: Master
Steps To reproduce--
1.  log in as an admin.
2.  Click on View All Notifications inside the Notifications on the top right corner.
Expected result: There should not be any UI issue on the notification page
Actual result: There is a UI issue on the notification page. Please find the screenshot Here

@prateek-webkul
Thanks, now this issue is fixed. so I am closing this issue.
And I have attached a screenshot for future reference Here


Third party and Server Side API's

Utilize and research the map api for geolocation
weather api for snow removal updates
square for payment systems
moment js or alternatives for curent time and future appts
etc

unnecessary issue


Will project live with new SSL security feautre?

Hi Dan. Could you please answer, will this project develop further after "certficate pinning" added to EFT?
Are there any ways to continue research API?

EFT already has certificate pinning.


where is remote_key

I have a problem.I only see power-key and four hdmi-key in the iphone home.where is other remote-key.need apple TV remote?

Yes, the other stuff is available through Apple Remote app, although it's not intuitive to find (I guess, it's intuitive if you use remote with Apple TV, remote app looks close to physical Apple TV remote). Volume control though is done by physical volume buttons, no on-screen UI is displayed. I haven't found how to mute, physical mute switch does not seem to mute stuff.

Thank you


Use relative paths in tutorials and examples

Our tutorials like https://graphstorm.readthedocs.io/en/v0.3.1/tutorials/quick-start.html are using absolute paths, assuming they are being executed from within the GS Docker container.
When referring to files in the repository we should be using paths relative to the root repo directory, to improve UX and allow users to directly copy-paste code from the guides.

To reduce confusion for readers of our documentations, we will unify the path to all graphstorm's artifacts, e.g., example codes, nodebooks, and yaml/json files, to be path_to_graphstorm/, and add a line of command cd path_to_graphstorm/ before any commands that refer to the relative path.
And then, all commands will be relative to this path, e.g. the command
python3 /graphstorm/examples/acm_data.py \,
will become,
python3 examples/acm_data.py \.


Change autocomplete behavior to replace user input with complete match to support more pattern match method.

Current autocomplete behavior uses append, which restricts us to case-sensitive prefix matching.
Changing appending to replacing allows us to implement any matching method we want.
For example:
class CodeAsmPrompt extends CodePrompt {
//...
@override
  bool match(String input) {
    // Modify this to use case-insensitive subsequence matching.
    return word != input && isSubsequence(word.toLowerCase(), input.toLowerCase());
  }
//...
}

We can also combine prefix matching and subsequence matching, similar to how VS Code does it.
I may have overlooked some boundary conditions; please review and let me know if there are any.
Another off-topic question: Can we expose _DefaultCodeAutocompleteListView to users? I think this ListView is good enough for users to just use it. Implementing my own ListView would require a lot of code to copy and paste.

Oh, I forgot to mention the highlights of _DefaultCodeAutocompleteListView.
Since it’s not exposed yet, I’ll add them after your feedback.

Thanks for the submission, I'll pass this PR. Looks like there will be some breaking changes in CodeAutocompleteResult, but there's no better way.
Another question, the match algorithm is implemented by CodeAutocompletePromptsBuilder and CodePrompt, and we have a default algorithm in _DefaultCodeAutocompletePromptsBuilder. I think the interfaces provided now are sufficient to cover user-defined matching scenarios.


Python 3 compatibility

Fix string compatibility between Python 2 and 3
Python 3 removed the decode() function from the string module, because all strings are unicode in Python 3.
This unbreaks sphinxcontrib/email.py for Python 3.
https://bitbucket.org/birkenfeld/sphinx-contrib/pull-requests/129/email-python-3-compatibility/diff

@s0undt3ch ?

I no longer maintain this extension nor do I have write perms to this repo, so, I really can't merge and release this.

@s0undt3ch I can grant you write permissions, if you want them. If not, would you object if I gave someone else permission to pick up maintenance? See https://github.com/sphinx-contrib/github-administration/issues/7

No objection to having someone else maintain the extension.

Closing this in favor of #2


Utilize DSIGNAlgorithm Ed25519Bip32DSIGN for extended keys in cardano-api

At the moment, Shelley extended signing keys are serialized using the cardano-crypto 128 byte binary format. However, these extended signing keys can actually be serialised to a more compact 96 byte format (see the DSIGNAlgorithm Ed25519Bip32DSIGN implementation).
The DSIGNAlgorithm Ed25519Bip32DSIGN implementation allows:

Serialisation of BIP32-Ed25519 extended signing keys to a more compact 96 byte format as opposed to the old cardano-crypto 128 byte format.

Deserialisation of BIP32-Ed25519 extended signing keys in either of the 96 or 128 byte formats.

@newhoggy @Jimbo4350 @erikd @dcoutts: When someone gets a chance, could I get an updated review on this?


Binding-specific autoStartup property has no effect (kafka bindings)

Setting the autoStartup=false property on specific bindings has no effect. The Kafka consumers still start.
Setting spring.cloud.stream.default.consumer.autoStartup=false all bindings are stopped, but cannot be activated individually (e.g. setting spring.cloud.stream.kafka.bindings.my-binding.autoStartup=true or spring.cloud.stream.bindings.my-binding.autoStartup=true).
I also raised a question on StackOverflow: https://stackoverflow.com/questions/73468885/start-binding-in-paused-state since I am not sure if I am just using it wrong. But following the documentation, I don't see what I am doing wrong.
I am using SpringBoot 2.7.4 with org.springframework.cloud:spring-cloud-dependencies:2021.0.4

@dinosaur-guy Just updated the SO thread with an answer - sorry for not replying earlier.
https://stackoverflow.com/questions/62577223/how-do-i-start-spring-cloud-stream-binders-manually-if-autostartup-is-set-to-fal
https://stackoverflow.com/questions/58795176/stop-consume-message-for-stream-listener
Basically, you need to use spring.cloud.stream.bindings.<binding-name>.consumer.autoStartup and set it to false or true to individually control it.
If you set that to false, then you need to use the BindingsEndpoint mechanism described in the above threads to start the binding.

Thank you. With those deeply nested configurations it is sometimes not easy to place the property at the correct place. This is working now.

Thanks for the feedback. Closing the issue since it is now working for you.


feat: Save network to sessionStorage

What it solves
Resolves #2921
How this PR fixes it
The current network is also saved in the sessionStorage.
How to test it
Open the /welcome route, change the network and refresh. The network should remain the same.

Would using in also work? https://stackoverflow.com/a/43805090/352796

Sadly not, TypeScript isn't a fan of conditionals involding unknown types: @katspaugh

Looks good to me
Reloads keeps me in the same network with no issues


Analyze Tag Usage Frequency

It would be good to have an ordered list of tags (key + val) used in ways, such that we can prioritize support.
We can then make milestones tracking our coverage of all tags used in >1% of ways, then 0.1% of ways, etc.
e.g. https://taginfo.openstreetmap.org/tags/oneway=yes is used in 1.77% of ways.

https://taginfo.openstreetmap.org/keys/highway#combinations
> 1%

[ ] name
[ ] surface
[ ] oneway
[ ] maxspeed
[ ] lanes
[ ] access
[ ] ref
[ ] tracktype
[ ] lit
[ ] bicycle
[ ] foot
[ ] bridge
[ ] footway
[ ] sidewalk
[ ] smoothness

> 0.1%

[ ] public_transport
[ ] width
[ ] bus
[ ] motor_vehicle

Should we exclude name and ref from this list? I can't see a very direct reason they'd influence current osm2lanes output, or a reason why to plumb them along in the output.

I was thinking that client using this library may just want to say "tell me
about way 12345". It's very little effort on our side and may remove the
need to parse tags on their side.
We also know that name and ref apply to all lanes.
On Sun, 8 May 2022, 13:49 Dustin Carlino, @.***> wrote:

Should we exclude name and ref from this list? I can't see a very direct
reason they'd influence current osm2lanes output, or a reason why to plumb
them along in the output.
—
Reply to this email directly, view it on GitHub
https://github.com/a-b-street/osm2lanes/issues/175#issuecomment-1120403795,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACEL2Z73RFK6DPR4DYIKOBDVI6S53ANCNFSM5U2BTHQQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Similarly, lit and bridge and other attributes are common to the entire road. Maybe we should plumb through things like this, or maybe we should hand people the Tags too? Otherwise it could be scope creep to have fields in the struct to represent everything

We could process the relevant tags like we do now, passthrough known tags
like name and ref, and then the remaining ones are in a "remainder" Tags
object. This solves another problem we have: how to have the website's
roundtrip system output tags you can actually paste back into osm without
losing information.
We could also use an input config decide if the output Tags should have
all tags or just the unprocessed remainders.
On Sun, 8 May 2022, 14:24 Dustin Carlino, @.***> wrote:

Similarly, lit and bridge and other attributes are common to the entire
road. Maybe we should plumb through things like this, or maybe we should
hand people the Tags too? Otherwise it could be scope creep to have
fields in the struct to represent everything
—
Reply to this email directly, view it on GitHub
https://github.com/a-b-street/osm2lanes/issues/175#issuecomment-1120409442,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACEL2Z72LBAUN7QDEDBKURDVI6XABANCNFSM5U2BTHQQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

What a great idea! I have been thinking to myself "is tag X actually combined with tag Y in the wild, or can we skip support for the combo". The taginfo combinations tab is a great resource, and I've been learning how to use OverpassQL properly, to answer questions like that too.


Read only user cannot view policies page

Description
Current Behavior
An error occurs as  " Error occurred while retrieving policy" when read only user access policies page.  According to the logs it seemed that read only users scope is set for policies.
This worked properly after adding internal/observer role was set to the apim:common_operation_policy_view in adavanced configurations.
Currently 'Add new policy' button is active for read only users
Desired behavior
Read only user should be able to view api policies.
'Add new policy' button should be disabled for read only users

Steps to Reproduce
prerequisites: create an API

Log in to the publisher portal as a read only user
Navigate to API -> policies

Affected Component
APIM
Version
4.1.0
Environment Details (with versions)
Local Environment
Relevant Log Output
No response
Related Issues
No response
Suggested Labels
No response

This error is fixed in 4.3.0 version. In API manager 4.3.0 read only users cannot save any changes to the polices they make.

This behaviour can demonstrate by logging to the portal as a read-only user as bellow.


python: Set indents to 4-space

Signed-off-by: Ryan Northey<EMAIL_ADDRESS>Commit Message: python: Set indents to 4-space
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

@htuch after a bit of messing around i just let yapf reformat to 4-space indents, so hopefully this one is pretty straight forward

even if its huge, sorry

the only inconsistency im seeing is the indents on docstrings
on the basis that this is the (reproducible) yapf reformatting and because of its size i would suggest we just apply this, and then im happy to go through and fix the indents on the docstrings

/retest


[FEATURE] submodel repository - Adding modular support for semantic models

Problem:
The component submodel repository lets you store a submodel. However, it only accepts very limited attributes:
https://aas-core-works.github.io/aas-core-meta/v3/Submodel.html
If I know want to store a battery pass submodel for example with performance data according to standard, the api removes any additional information, storing only the predefined attributes.
I would like:
Since the standards for semantic models are already defined with their constraints, I would like to add semantic models like this  for validation and ultimately storing them in a database.

Hello @mmaeffert,
could you explain which data is actually lost from the performance data? Are you talking about the PerformanceEntity from the example AASX-file?
Regarding the last point from your issue:
Do you want to store this schema using the submodel repository with mongoDB as backend so that it's saved in a database?

@aaronzi Error was on my end

Alright, nice to hear that you found the problem


Inputs fix

Summary

Addresses issues with inputs #869
Updated styling of disabled buttons
Updated padding of grids containing inputs to accommodate new input margin

@tananaev Kindly let me know if I've missed any part the inputs were not showing correctly

Nice catch. Will address it

@tananaev done

Merged, thank you.


DOP-4247: New Github Action to build docs site artifact

Ticket
DOP-4247
Notes
This new custom Github Action is created to download build artifacts of a docs site so that later in the pipeline another Action can run a Lighthouse score on the built site.
It fetches the most recent build data from the Snooty Data API per the project designated, parses the data apart into a similar structure as our zipped manifests, and saves them to the process for Snooty to use in a build.
There is a companion PR in Snooty to be opened soon.
Behavior with both PRs
Cloud-docs
Server Docs
^ note above you can download the build artifacts

minor. theres a few typescript errors from the CI / TS tests. most are fixable with vscode formatter using eslint but some are not automatic


Any way to get 4xx response body w/o try...catch while using await?

async function checkToken(token) {
  const result = await superagent
    .post(`${config.serviceUrl}/check_token`)
    .send({token});
  return result.body;
}

Default options if this call will return 401 throws an Exception, which is not what i expect. API i call is using HTTP status messages also as body to give information back and i just need the body part.
Response with http status 401 is
{
    "data": null,
    "error": {
        "code": "INVALID_TOKEN",
        "message": "Token is not valid"
    }
}

And currently, to get this, I need to wrap all superagent calls with try...catch
async function checkToken(token) {
  let result = null;
  try {
    result = await superagent
      .post(`${config.serviceUrl}/check_token`)
      .send({token});
  } catch (e) {
    result = e.response;
  }
  return result.body;
}

Any way to have the 1st sample working and returning JSON w/o looking at HTTP status?

There's .ok() callback which lets you decide which responses are considered sucessful.

In case if anyone else would found this issue:
In Jest you can
const response = superAgent.get(url)
await expect(response).rejects.toMatchObject({
  status: 400,
  response: {
    text: '[expected error message]'
  }
})

Code for Chai or Should.js would be very similar


Device did not appear with the intermediate firmware - Moes WS-EU1-LB 1 Gang No Neutral Switch

Hi,
I purchased one of these Moes 1 Gang No Neutral switch which i checked and was compatible with tuya-convert. I've looked inside and it is running an ESP8285 chip however I have been unable to flash the firmware using tuya-convert.
I'm a relative newby so have been following the various videos on youtube closely. I've tried using both iphone & android tablet for the flashing as I read that sometimes there are issues when using a iOS device.
Below are logs:
smarthack-web.log
smarthack-udp.log
smarthack-wifi.log
smarthack-mqtt.log
smarthack-psk.log
Any help would be gratefully received.
Thanks

got a Yagusmart No-neutral 3, 2, and 1 gang, it looks exactly the same as the Moe's line (im assuming its the same just marketed under a different name). I was able to flash all 4 of my 3 gang sand configure them using the Moes no-neutral 3-gang template and it worked perfectly.
Problem is, for some reason im only able to do that with the 3 gangs, for the life of me i couldnt manage to flash the 1 gang and 2 gang, i get the same error as @jimmi1982
note: i am an utter noob at this,  not even sure if i should be posting this here, its a long shot but im hoping its the same issue
any help or direction would be amazing, im clearly way out of my league and no idea where to start researching.
thank you in advance!

Duplicate of #483


Set segment version to v3

Is PBNJ directly generate v3 format segment and push it to controller? That will cause a lot of problem because a lot of code assume that segments on controller are in v1 format.

Codecov Report

Merging #1708 into master will increase coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1708      +/-   ##
==========================================
+ Coverage   67.87%   67.94%   +0.06%
==========================================
  Files         830      830
  Lines       40388    40388
  Branches     5313     5313
==========================================
+ Hits        27415    27440      +25
+ Misses      11114    11100      -14
+ Partials     1859     1848      -11

Impacted Files
Coverage Δ

.../linkedin/pinot/transport/common/SegmentIdSet.java
53.84% <0%> (-5.13%)
:arrow_down:

...filter/SortedInvertedIndexBasedFilterOperator.java
77.02% <0%> (-2.71%)
:arrow_down:

...manager/realtime/HLRealtimeSegmentDataManager.java
74.03% <0%> (-2.21%)
:arrow_down:

...not/broker/requesthandler/RangeMergeOptimizer.java
88.7% <0%> (-1.62%)
:arrow_down:

...ot/core/query/pruner/ColumnValueSegmentPruner.java
87.93% <0%> (ø)
:arrow_up:

...manager/realtime/LLRealtimeSegmentDataManager.java
70.13% <0%> (+0.19%)
:arrow_up:

.../broker/routing/HelixExternalViewBasedRouting.java
90.94% <0%> (+0.34%)
:arrow_up:

...pby/NoDictionarySingleColumnGroupKeyGenerator.java
92.92% <0%> (+1.01%)
:arrow_up:

.../pinot/core/common/datatable/DataTableBuilder.java
92.17% <0%> (+1.11%)
:arrow_up:

...lix/core/realtime/PinotRealtimeSegmentManager.java
72.31% <0%> (+1.12%)
:arrow_up:

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8cc6b7f...f21772a. Read the comment docs.


Service Health Alerts howto guidance out of date

Document is out of date - the document refers to adding Service Health alerts as part of Activity Log alerts, which is not correct
Step 3 on this document:
Select +Add activity log alert, and set up an alert to ensure you are notified for future service notifications.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 37d1c7cf-1642-242c-a712-fd5c5c707173
Version Independent ID: ba4c04cd-8a55-32f1-e75f-874951b51c35
Content: View service health notifications by using the Azure portal - Azure Service Health
Content Source: articles/service-health/service-notifications.md
Service: service-health
GitHub Login: @rboucher
Microsoft Alias: robb

@vanessabruwer Thanks for the feedback !  I have assigned this issue to content author to investigate and update the document as appropriate.

Thanks for your dedication to our documentation. This issue has been moved to an internal work item for triage and prioritization. Thanks in advance for your understanding as we investigate to provide the most accurate documentation updates. #please-close
Internal work item: https://dev.azure.com/msft-skilling/Content/_workitems/edit/271112
Please also note that GitHub documentation issues are being phased out completely. For more information, see: https://aka.ms/ContentUserFeedback.
Our support team is available to assist you if you have any pressing questions or problems that need to be resolved. You can create a support ticket by visiting the following link: https://azure.microsoft.com/support/create-ticket/
You may also ask support questions in our Q&A forums.


Extend Script doesn't work with script without class_name

Godot version
4.0.beta.custom_build (c0de8d32d)
System information
Windows 11, Vulkan Clustered, Geforce 1660 (522.25)
Issue description

Extend Script does not create an inheritor script. After clicking Create button, the following appears in the console:
editor\script_create_dialog.cpp:379 - Parameter "type" is null.

Steps to reproduce

add script (without class_name) to node
extend it with RMB->Extend Script

Minimal reproduction project
No response

Duplicate of #65827


fix: Fix android permission bug

This PR fixes a bug for Android permissions.

Android does not have an undetermined permission state
The check for the permission state wrongly returned undetermined for a denied state
There was also a bug with the permission listener not being unset after it finished

Fixes #860

Thanks @navaronbracke. As today 7/30/2024, using 5.1.1 version, it is still an issue.