setting MKL pretrained subkernel weights

Hi all.
Im trying to figure out how to use a vector of pretrained subkernel weights for the MKL module. Im trying to understand this post (script):
http://shogun-list.shogun-toolbox.narkive.com/Ff1tmKJM/set-subkernel-weights-does-not-work-in-python-modular-interface
however, in the mentioned script, Im not sure how mkl.classify() works without previously executing mkl.set_kernel(kernel), given kernel was modified by kernel.set_subkernel_weights(w, 3). Can someone explain this segment of the script or point me to some other clearer one where pretrained weights are loaded and applied?
Thank you very much...

Hi @iarroyof. Are you obtaining the weights using Shogun or from somewhere else? In case you are also using Shogun for the training, maybe using serialization solves your problem. See for instance serialization_string_kernels_modular.py or other serialization examples.

Hi @iglesias thank you for reply. Actually mi question is in the sense Im not sure what is the procedure to update que subkernel weights of a combinedKernel and then, applying this update to new data via a MKL object. My MKL Shogun object is already trained. Its subkernel weights are stored into a file. They can be retrieved at any time as a python list.
I'd like to instantiate a completely new MKL object, in such a way transferring the stored (pretrained) weights to it is possible. I'd like this new MKL object can work just like a newly trained one (but using pretrained weights instead).

Hi again... I have tried with this code:
from modshogun import *
from tools.load import LoadMatrix
from mklObj import genKer
from numpy import array

[traindata,
  testdata,
  trainlab,
  testlab] = load_Toy('/home/iarroyof/shogun-data/toy/',	# Data directory
		       'fm_train_multiclass_digits500.dat',		# Multiclass dataSet examples file name
		       'label_train_multiclass_digits500.dat')	# Multiclass Labels file name

feats_train = RealFeatures(traindata) 	# train examples
labelsTr = MulticlassLabels(trainlab)	# train multiclass labels
feats_test = RealFeatures(testdata)  	# test examples
labelsTs = MulticlassLabels(testlab)	# test multiclass labels

# Parameters
kernelFamily = 'gaussian'
sigmas = [10, 20, 70, 120]
weights = array([0.1, 0.1, 0.3, 0.5])

# Setting up the MKL machine
mkl = MKLMulticlass()
# Generating a combinedKernel
kernel = genKer(featsL = feats_train, featsR = feats_test, basisFam = kernelFamily, widths = sigmas)
kernel.init(feats_train, feats_test)
kernel.set_subkernel_weights(weights)      # Setting pretrained weights
mkl.set_kernel(kernel)
mkl.set_labels(labelsTs)

out = mkl.apply()   # Applying
# Evaluating the test performance
evalua = MulticlassAccuracy()
testerr = evalua.evaluate(out, labelsTs)
print 'Kernel evaluation ready. The precision was: ', testerr*100, '%'

After trying to run, the following error is given:
Traceback (most recent call last):   File "mkl2.py", line 81, in <module>     out = mkl.apply()                      SystemError: [ERROR] In file /home/iarroyof/shogun/src/shogun/machine/MulticlassMachine.cpp line 113: num_machines = 0, did you train your machine?

Hi... I have kept trace of this problem. By testing different situations with the code of my previous post I have inferred the way is loading a pretrained MKL machine (please correct me if it not (too) necessary). In this direction, I have tested  MKLClassification.save() by using the next code snippet:
# ... MKL object instantiation, parameter setting, kernel generation and initialization...
mkl.train()
f = open('mkl_saved.mkl', 'w')
mkl.save(f)

mkl.load(f)
sigmas = [10, 20, 70, 120]
weights = array([0.1, 0.1, 0.3, 0.5])
kernel = genKer(featsL = feats_train, featsR = feats_test,
                        basisFam = kernelFamily, widths = sigmas)
kernel.set_subkernel_weights(weights)
kernel.init(feats_train, feats_test)
mkl.set_kernel(kernel)
mkl.set_labels(labelsTs)
out = mkl.apply()

However, the next error is given:
Traceback (most recent call last):
  File "mkl2.py", line 56, in <module>
    mkl.save(f)
TypeError: in method 'CSVM_save', argument 2 of type 'FILE *'

Which made me to think there is a problem with the Shogun Python wrapping, so I review the implementation of save(). I noted it is needed uniquely saving some specific components of the MKL inner SVM. Again, necessary methods for retrieving these components seem to be also unavailable for the Python interface.

All right, let's try to go one part at a time.

I am not familiarised with the implementation of MKL so perhaps it is actually required to call train in an MKL object before you can call apply. Maybe some internal trained state is set upon call to train and it is checked before calling apply, failing the call if the trained state has not been set.

Would cloning the MKL trained object solve your problem?
The C++ example you have cited above is not the same as what you are doing in your Python code. Note that in the C++ example tsvm has been trained before setting the kernel.

I am unsure about they way you are using save to serialize. The serialization Python examples and also in some code I used in the past I used pickle.

Thank you @iglesias, I will try by serializing...
PD. The code I used is updated...

Keep us updated about your progress and feel free to close the issue once you consider it solved.

Thank you @iglesias. Do know if this code snippet is correct?:
mkl.train()
f = open('mkl_saved.mkl', 'w')
mkl.save(f)

output: TypeError: in method 'CSVM_save', argument 2 of type 'FILE *'
or it is needed convert f to another data type, before passing it to the mkl save() method.

Hi... I reimplemented saving and loading method for Python. Here is an informal test code:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from modshogun import *
from tools.load import LoadMatrix
from mklObj import genKer
from mklObj import load_binData
from numpy import *

# Loading toy data from files
[feats_train, feats_test, labelsTr, labelsTs] = load_binData(tr_ts_portion=0.75,
                fileTrain='fm_ape_gutXX.dat',		# two-class dataSet examples file name
                fileLabels='bin_label_ape_gut.dat')	# two-class Labels file name

# Parameters
kernelFamily = 'gaussian'
mkl.set_C(1,1)
mkl.set_mkl_norm(2)
sigmas = [10, 20, 70, 120] # toy widths

# =============================== Training =============================================================
# some function for creating a one-time combined kernel
kernel = genKer(featsL = feats_train, featsR = feats_train, basisFam = kernelFamily, widths = sigmas)
kernel.init(feats_train, feats_train)

# Setting up the MKL machine
mkl = MKLClassification()           # MKL object
mkl.set_kernel(kernel)
mkl.set_labels(labelsTr)
mkl.train()
# Note: Do not to include sigmas because they are kernel family dependent
def save_mkl(mkl, kernel, file_name): # Function for saving a pretrained MKL machine
    mkl_machine = {}
    support=[]
    mkl_machine['num_support_vectors'] = mkl.get_num_support_vectors()
    mkl_machine['bias']=mkl.get_bias()
    for i in xrange(mkl_machine['num_support_vectors']): # whole vector setting methods (e.g. set_alphas()) have
                                                         # problems, so we used individual setting methods and a
                                                         # for loop.
        support.append((mkl.get_alpha(i), mkl.get_support_vector(i)))
    mkl_machine['support'] = support
    mkl_machine['weights'] = list(kernel.get_subkernel_weights())

    f = open(file_name,'w')
    f.write(str(mkl_machine)+'\n')

save_mkl(mkl, kernel, 'mkl_saved.model') # Saving the pretrained MKL machine
del mkl
del kernel
# ================= Pretrained machine loading and tests ===========================================================
def load_mkl(file_name): # Function for loading a pretrained MKL machine.

    with open(file_name,'r') as pointer:
        mkl_machine = eval(pointer.read())

    mkl = MKLClassification()
    mkl.set_bias(mkl_machine['bias'])
    mkl.create_new_model(mkl_machine['num_support_vectors'])
    for i in xrange(mkl_machine['num_support_vectors']):
        mkl.set_alpha(i, mkl_machine['support'][i][0])
        mkl.set_support_vector(i, mkl_machine['support'][i][1])

    return mkl, array(mkl_machine['weights'])

[mkl_copy, ws] = load_mkl('mkl_saved.model') # Loading the pretrained MKL machine from file. Now it is used test data.
kernel_copy = genKer(featsL = feats_train, featsR = feats_test, basisFam = kernelFamily, widths = sigmas)
kernel_copy.set_subkernel_weights(ws)
kernel_copy.init(feats_train, feats_test)

mkl_copy.set_kernel(kernel_copy)
out = mkl_copy.apply()

evalua = ErrorRateMeasure()     # performance measures are computed.
testerr = 100 - evalua.evaluate(out, labelsTs) * 100
print testerr

In this way, think the issue is solved

Hi... I reimplemented saving and loading methods for Python. This implied saving not only pretrained weights but also components of the trained MKL model. These elements can be verified in the body of the save_mkl() method I have implemented to backup such a training model. Here is an informal test code:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from modshogun import *
from tools.load import LoadMatrix
from mklObj import genKer
from mklObj import load_binData
from numpy import *

# Loading toy data from files
[feats_train, feats_test, labelsTr, labelsTs] = load_binData(tr_ts_portion=0.75, # train data portion
                                                                 fileTrain='toy_data.dat',     # Two-class examples file name
                                                                 fileLabels='toy_labels.dat')	# Two-class Labels file name
# Parameters
kernelFamily = 'gaussian'
sigmas = [10, 20, 70, 120]

# =============================== Training ====================================
kernel = genKer(featsL = feats_train, featsR = feats_train, basisFam = kernelFamily, widths = sigmas)
kernel.init(feats_train, feats_train)
# Setting up the MKL machine
mkl = MKLClassification()           # origin MKL object
mkl.set_C(1,1)
mkl.set_mkl_norm(2)
mkl.set_kernel(kernel)
mkl.set_labels(labelsTr)
mkl.train()                         # Train the origin machine
# Note: We preferred discarding sigmas from this file because they are kernel family dependent
def save_mkl(mkl, kernel, file_name): # Python reimplementated function for saving a pretrained MKL machine.
    mkl_machine = {}
    support=[]
    mkl_machine['num_support_vectors'] = mkl.get_num_support_vectors()
    mkl_machine['bias']=mkl.get_bias()
    for i in xrange(mkl_machine['num_support_vectors']):
        support.append((mkl.get_alpha(i), mkl.get_support_vector(i)))

    mkl_machine['support'] = support
    mkl_machine['weights'] = list(kernel.get_subkernel_weights())

    f = open(file_name,'w')
    f.write(str(mkl_machine)+'\n')
    f.close()

save_mkl(mkl, kernel, 'mkl_saved.model') # Saving the dictionary containing trained MKL machine into a file
del mkl

# ================= Pretrained machine tests =======================================
def load_mkl(file_name): # Python reimplemented function for loading a pretrained MKL machine.
    with open(file_name, 'r') as pointer:
        mkl_machine = eval(pointer.read())

    mkl = MKLClassification()           # A new two-class MKL object
    mkl.set_bias(mkl_machine['bias'])
    mkl.create_new_model(mkl_machine['num_support_vectors']) # Initialize the inner SVM
    for i in xrange(mkl_machine['num_support_vectors']):
        mkl.set_alpha(i, mkl_machine['support'][i][0])
        mkl.set_support_vector(i, mkl_machine['support'][i][1])

    return mkl, array(mkl_machine['weights'])

[mkl_copy, weights] = load_mkl('mkl_saved.model') # Loading the pretrained MKL machine. We now use test data.
kernel_copy=genKer(featsL = feats_train, featsR = feats_test, basisFam=kernelFamily, widths = sigmas)
kernel_copy.set_subkernel_weights(weights)        # Setting up pretrained weights to the new kernel
kernel_copy.init(feats_train, feats_test)
mkl_copy.set_kernel(kernel_copy)

out = mkl_copy.apply()                            # Test the loaded machine

evalua = ErrorRateMeasure()     # performance measures are computed.
testerr = 100 - evalua.evaluate(out, labelsTs) * 100
print testerr

In this way, I think the issue is solved only for two-class problems. I will try for regression model because I need it. For multiclass mkl object needed getter and setter methods weren't wrapped by Swig. I also confirmed mkl.save() as well as mkl.load() methods were not well wrapped by Swig (they showed errors like the mentioned in my previous post). Other needed mkl (two-class and multiclass) object methods have similar problems, e.g. mkl.set_alphas(), mkl.load_alphas(), mkl.set_support_vectors() and mkl.load_support_vectors(), etc.


[InputBase] Fix placeholder bug in Edge

[x] I have followed (at least) the PR section of the contributing guide.

Fixes #15183
This PR fixes a bug related to the Placeholder and Label. They are being overlapped when a Input is focused and then unfocused by clicking at the address bar.
This bug is only happening in Edge.
FYI @oliviertassinari :)

@rodrigolabs I have rebased the pull request on the next branch. We keep master for important bug fixes. This one is about polishing the details ✨.

@rodrigolabs It's a great first pull request on Material-UI 👌🏻. Thank you for working on it!

@oliviertassinari Thank you! It was a pleasure to contribute after a while just using Material-UI ❤


Update the sawtooth-default.yaml

Update the sawtooth default docker-compose
version to 2.0 since it does fails with 2.1
Fixes: #22
Signed-off-by: Swapnil Kulkarni<EMAIL_ADDRESS>

Check to make sure that you have Docker version 17.03.1-ce or greater, and Docker Compose version 1.9.0 or greater. These versions are required for the version 2.1 to work successfully.


Only run DFLOW if the peak flow is above a certain threshold

@dcadol
The threshold will be 555 cfs = 15.7159 cms (according to google, so pls check that math)

Yes, that's exactly right. The conversion factor is 1 cms = 35.315 cfs.
For simplicity, I think you can safely round the threshold value to just 16 cms.


refactor(over window): use new table.iter_with_prefix in streaming OverWindow

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
After https://github.com/risingwavelabs/risingwave/pull/12364 we now have a new and handy StateTable::iter_with_prefix method to iterate a given range within a pk prefix. This PR utilize it to simplify OverPartition. Fixes #12583.
Checklist

[x] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[x] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Do we need some perf test for this PR? IIRC, after using state table sub_range interface, over window will scan less items.

Actually we don't have a good benchmark query specifically for OverWindow yet. I'll run nexmark q6-group-top1 and q19, hope they can give some basic sense for the performance impact of this PR.


Scalatest log output disappears with Maven 3.5.0

Scalatest log output disappears with Maven 3.5.0, and I can't seem to get it back.  As much as I hate to see all that Parquet log vomit, it would be useful to see the unit test failure messages.
$ mvn --version
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T14:39:06-05:00)
Maven home: /usr/local/Cellar/maven/3.5.0/libexec
Java version: 1.8.0_111, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.6", arch: "x86_64", family: "mac"

$ mvn test -Dsuites=org.bdgenomics.adam.rdd.fragment.FragmentRDD
[INFO] Scanning for projects...
...
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ adam-core_2.10 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ adam-core_2.10 ---
*** RUN ABORTED ***
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ADAM_2.10 .......................................... SUCCESS [  3.749 s]
[INFO] ADAM_2.10: Core .................................... FAILURE [  5.117 s]
[INFO] ADAM_2.10: APIs for Java ........................... SKIPPED
[INFO] ADAM_2.10: CLI ..................................... SKIPPED
[INFO] ADAM_2.10: Assembly ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.013 s
[INFO] Finished at: 2017-04-18T10:13:05-05:00
[INFO] Final Memory: 34M/728M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test)
on project adam-core_2.10: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :adam-core_2.10

I think we should hold off on requiring Maven 3.5.0 then. ;)
What do you get with -X?

I think we should hold off on requiring Maven 3.5.0 then. ;)

Can't help it, brew gave it to me.

What do you get with -X?

[ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project adam-core_2.10: There are test failures -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project adam-core_2.10: There are test failures
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures
	at org.scalatest.tools.maven.TestMojo.execute(TestMojo.java:107)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
	... 20 more

It appears that things are broken only when -Dsuites is provided.

Closing as NotOurProblem for now.  Development on scalatest-maven-plugin looks to be completely dead, so I think we'll just have to DealWithIt.


CAMEL-20377: fixes a confusing typo in the pom file

Fixes a typo originally found on CAMEL-20337 by Craig Ziesman<EMAIL_ADDRESS>

@oscerd @davsclaus This is the only part we need from #12966. It doesn't really cause any problem, but it can be confusing. It would be good to have this on 4.4.0.


"alias" unexpected behaviour.

Hi,
I was under the naive assumption that the alias definition would allow me to call the application directly on rez-env invokation:
>>>>>>>> C:\Users\thomas>rez-env nuke-9.0v8 -- nuke
'nuke' is not recognized as an internal or external command,
operable program or batch file.

if I launch a shell and then invoke nuke it is fine though:
>>>>>>> C:\Users\thomas>rez-env nuke-9.0v8 -- cmd
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

>>>>>>>> C:\Users\thomas>nuke
Nuke 9.0v8, 64 bit, built Oct 26 2015.
Copyright (c) 2015 The Foundry Visionmongers Ltd.  All Rights Reserved.
Disk cache C:/Users/thomas/AppData/Local/Temp/nuke\ViewerCache/??: 10065MB (99% of 10240MB) used in 1042 files.

This is my package.py file by the way:
# -*- coding: utf-8 -*-

name = 'nuke'
version = '9.0v8'
authors = ['The Foundry Visionmongers Ltd.']
description = 'Powerful compositing, editorial and review.'
tools = ['nuke', 'nukex']  # Testing stuff here.
uuid = 'third-party.the-foundry-visionmongers.nuke'

def commands():
    import os
    import platform

    application_name = '{0} {1}-{2}'.format(
        'The Foundry Visionmongers Ltd.', this.name.title(), this.version)
    error_message = '"{0}" is not available on your platform!'.format(
        application_name)

    if platform.system() == 'Darwin':
        raise SystemError(error_message)
    elif platform.system() in ('Windows', 'Microsoft'):
        root_directory = 'C:\\Program Files\\{0}{1}'.format(
        this.name.title(), this.version)
    else:
        raise SystemError(error_message)

    assert os.path.exists(root_directory), (
        '"{0}" is not installed at a standard location on your system!'.format(
        application_name))

    # Setting aliases after *PATH* manipulation will break the aliases command:
    # https://github.com/nerdvegas/rez/issues/343
    alias('nuke', "Nuke9.0")
    alias('nukex', "Nuke9.0 --nukex")

    env.PATH.append(root_directory)

Cheers,
Thomas

Thanks Thomas - I would expect this to work, will need to investigate.
Also, unrelated, but we're looking into a new feature that will fix issue
#343, which I see you have noted in your package.py.
Thx
A
On Thu, Feb 23, 2017 at 10:00 AM, Thomas Mansencal<EMAIL_ADDRESS>
wrote:

Hi,
I was under the naive assumption that the alias definition would allow
me to call the application directly on rez-env invokation:

C:\Users\thomas>rez-env nuke-9.0v8 -- nuke'nuke' is not recognized as an internal or external command,
operable program or batch file.

if I launch a shell and then invoke nuke it is fine though:

C:\Users\thomas>rez-env nuke-9.0v8 -- cmd
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\thomas>nuke
Nuke 9.0v8, 64 bit, built Oct 26 2015.
Copyright (c) 2015 The Foundry Visionmongers Ltd.  All Rights Reserved.
Disk cache C:/Users/thomas/AppData/Local/Temp/nuke\ViewerCache/??: 10065MB (99% of 10240MB) used in 1042 files.

This is my package.py file by the way:
-- coding: utf-8 --
name = 'nuke'
version = '9.0v8'
authors = ['The Foundry Visionmongers Ltd.']
description = 'Powerful compositing, editorial and review.'
tools = ['nuke', 'nukex']  # Testing stuff here.
uuid = 'third-party.the-foundry-visionmongers.nuke'
def commands():
import os
import platform
application_name = '{0} {1}-{2}'.format(
    'The Foundry Visionmongers Ltd.', this.name.title(), this.version)
error_message = '"{0}" is not available on your platform!'.format(
    application_name)

if platform.system() == 'Darwin':
    raise SystemError(error_message)
elif platform.system() in ('Windows', 'Microsoft'):
    root_directory = 'C:\\Program Files\\{0}{1}'.format(
    this.name.title(), this.version)
else:
    raise SystemError(error_message)

assert os.path.exists(root_directory), (
    '"{0}" is not installed at a standard location on your system!'.format(
    application_name))

# Setting aliases after *PATH* manipulation will break the aliases command:
# https://github.com/nerdvegas/rez/issues/343
alias('nuke', "Nuke9.0")
alias('nukex', "Nuke9.0 --nukex")

env.PATH.append(root_directory)

Cheers,
Thomas
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nerdvegas/rez/issues/389, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABjqSrEN7zpGMHdQw_TtOOliyGqWtlpZks5rfL4JgaJpZM4MJTzG
.

@nerdvegas: Right, glad to see that my assumptions were correct! Looking forward to the fix.

This seems to be a different issue but i just wanted to raise awareness, that alias breaks if called after any path manipulation including other packages. So if you have a set of packages and others are resolved before nuke and touch PATH, then PATH is emptied and standard commands do not work till after the resolve when the original PATH is re-appended.
Also, keep in mind, that "original" is not the PATH in the shell you start rez-env from, but taken from the registry on windows (which i would like to see changed).

Hey Thorsten,
Thanks, yeah this is a separate issue and one that I'm aware of. I think
this is being worked on atm, will be checking in in the coming weeks.
Thx
A
On Fri, Feb 24, 2017 at 7:52 AM, Thorsten Kaufmann<EMAIL_ADDRESS>
wrote:

This seems to be a different issue but i just wanted to raise awareness,
that alias breaks if called after any path manipulation including other
packages. So if you have a set of packages and others are resolved before
nuke and touch PATH, then PATH is emptied and standard commands do not work
till after the resolve when the original PATH is re-appended.
Also, keep in mind, that "original" is not the PATH in the shell you start
rez-env from, but taken from the registry on windows (which i would like to
see changed).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nerdvegas/rez/issues/389#issuecomment-282117496, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABjqSm1u3UKMfDxlw0QqCnygVJ4-_XHZks5rffGhgaJpZM4MJTzG
.


I created a WordPress version

Hi Nick
I love this simple script so much I created a WordPress plugin based on it to allow those a little less technical to use your creation.
I will be adding more features as I progress, it can be viewed at https://wordpress.org/plugins/shareon/
Regards

Hi Gareth, this looks great! Thank you for porting this extension. I'll include the link to it in the docs in the near future


Build compilation problem

Hi,
I get the following errors during my build:
/GeoCoderTest/android/app/src/main/java/com/geocodertest/MainActivity.java:16: error: cannot find symbol
    protected List<ReactPackage> getPackages() {
              ^
  symbol:   class List
  location: class MainActivity
GeoCoderTest/android/app/src/main/java/com/geocodertest/MainActivity.java:16: error: cannot find symbol
    protected List<ReactPackage> getPackages() {
                   ^
  symbol:   class ReactPackage
  location: class MainActivity
/GeoCoderTest/android/app/src/main/java/com/geocodertest/MainActivity.java:15: error: method does not override or implement a method from a supertype
    @Override
    ^
GeoCoderTest/android/app/src/main/java/com/geocodertest/MainActivity.java:18: error: cannot find symbol
          new MainReactPackage(),
              ^
  symbol:   class MainReactPackage
  location: class MainActivity
/GeoCoderTest/android/app/src/main/java/com/geocodertest/MainActivity.java:17: error: cannot find symbol
        return Arrays.<ReactPackage>asList(
                       ^
  symbol:   class ReactPackage
  location: class MainActivity
/GeoCoderTest/android/app/src/main/java/com/geocodertest/MainActivity.java:17: error: cannot find symbol
        return Arrays.<ReactPackage>asList(
               ^
  symbol:   variable Arrays
  location: class MainActivity
6 errors
:app:compileDebugJavaWithJavac FAILED

FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:compileDebugJavaWithJavac'.

I tried importing java.util.Arrays, java.util.List, However I was not sure of the ReactPackage and MainReactPackage classes and also the @Override annotation.
Looking forward for a feasible fix.

Can someone help with this? I'm getting the same build problems on this.

I think there is an error in the documentation, the RNGeocoderPackage goes in MainApplication.java, not MainActivity.java

Thank you pjb3, will give that a shot, the documentation is definitely wrong.


whitespace problem in transactions / let OSQL statements

Hi OrientDB team,
I need to insert strings that contain whitespace \n in transactions with LET, e.g.,
script sql
begin
LET a=insert into V set greeting="hello\norientdb"
commit
end

however, this throws (2.1-rc4):
Error: com.orientechnologies.orient.enterprise.channel.binary.OResponseProcessingException: Exception during response processing.

Error: com.orientechnologies.orient.core.sql.parser.TokenMgrError: Lexical error at line 1, column 34.  Encountered: "\n" (10), after : "\"hello"

Is it possible to insert strings with \n into OrientDB using transactions?

Hi @ycxq
to insert backslash character you have to quote it (double backslash), eg.
script sql
begin
LET a=insert into V set greeting="hello\\norientdb"
commit
end


Add benchmark for Sequence.reversed()

Add benchmark for Sequence.reversed(). Would be useful for #30324.

@swift-ci benchmark

@swift-ci benchmark

@palimondo, could you please re-run benchmark once again. Thank you.

ping @palimondo

@swift-ci benchmark

@valeriyvan I’m not sure what level of improvements you are getting locally from #30324, but try to set the base workloads in this new benchmark family so that the BenchmarkDoctor would not complain they are too small after you get the performance win you hope for, if possible.

@valeriyvan One more thought: the motivation in #30324 mentions speedup for case with underestimatedCount slightly less than count. Is that a necessary condition for the perf win? If it makes a significant difference, maybe we need to add such kind of sequence to the benchmark. All we have here either return 0 or exact count:


Gridsome g-image preload before view

Is it possible to preload g-image before it is in browser view?
Like this preload setting in VueLazyLoad:
Vue.use(VueLazyload, {
preLoad: 1.3
})
It will make it will make better flow, which won't always show the blur image. Because it's preloaded before it is in view, instead of always showing a blur image and waiting it to reveal it self.
Thanks

I think the blurring is best for g-image, because it allocates space within the layout so no elements are shifted around when the images do load.
However, correct me if I'm wrong, but couldn't you accomplish this with a plain <img> tag?

@theisws Perhaps :immediate="true" is what you need?
https://gridsome.org/docs/images/#image-cropping

Oh wow, didn't even realize that was an option.

@tkhquang, @Sergix thanks for replying.
But it doesn't seem to do the trick for me.
Using :immediate="true" loads all images on the page, which results in really bad Lighthouse score.
The blurry image isn't used, so the page will start "jumping" as the images loades.
If there somehow were a way, where images gets lazy-loaded på before they enter the screen (etc. one screenheight before) will make the scrolling and feeling more smooth, now it feels like you are waiting on an image to appear from the blurry effect.
As i've used VueLazyLoad before, with preLoad setting, you'll often not see the image lazyload, because it's often loaded before. That's really the case i want, i dont need to see blurry images, but it keeps the page-height, the best scenario will be never see a blurry image, but in an image-heavy website i know it's not possible. But there must be a way to make the user feeling more smooth, instead of scroll -> wait for image (repeat) all the way down.
Thanks

@theisws
I see, you might want to take a look at this line. I could be wrong but there's currently no user options for this to be set, so this looks like a feature request.
https://github.com/gridsome/gridsome/blob/accd1090a882bdbdbed1cdb29d7c5b67058db910/gridsome/app/directives/image.js#L22
For more information about the Intersection Observer API

@Sergix maybe you can change the label of this issue for "Feature Request"? I would love to have this option implemented on g-image 🙂

Likewise, this is a must-have feature.

+1 that this is a must-have feature.
In my project, I've applied an extremely ugly workaround but it works for me. Haven't noticed any unwanted side effects in my specific case. If you wish to use it, you must check whether IntersectionObserver is used throughout your project by any other libraries.
Add this to your App.vue script:
const OriginalIntersectionObserver = IntersectionObserver;
window.IntersectionObserver = (...args) => {
  const isGridsome = args.length === 2 && args[1] && args[1].rootMargin === '20px' && args[1].threshold === 0.1;
  return isGridsome
    ? new OriginalIntersectionObserver(args[0], Object.assign(args[1], { rootMargin: '200px' }))
    : new OriginalIntersectionObserver(...args);
};

@SorJEF really thank you for your workaround! Maybe @hjvedvik has any feedback to give in this feature request? 🚀


Vape is not saving when going into a game

I've used vape v4 for roblox before but now it just seems to not save when I go into a game anymore I use mobile I switched my executors but nothing fixed it the only time the settings save is in the lobby I disabled Lobby checking that did not work and I don't use the settings often

????????????????????

Whenever I load into a Lobby the settings I put there will save but whenever I load into a game in bedwars it will go back to default

cannot repro.


Why is blessed a dev dependency as opposed to a normal dependency?

If there's no real reason, can I send a PR to make blessed a normal dependency? I tried to npm install blessed-contrib but I couldn't run the example off the bat because I had to install blessed.

Hi @mjkaufer
blessed-contrib is sort of a plugin for blessed but does not directly depend on it. The example is not part of blessed-contrib core (if you install blessed-contrib you will not need the sample). That is why the README recommends to install blessed before running the sample. This is a good point though: blessed could be considered a peer dependency of blessed-contrib (http://blog.nodejs.org/2013/02/07/peer-dependencies/) so any PR on this is welcome!

+1 to PeerDependency, I have put them in both peerDependencies (for production) and devDependencies (for testing) without problems.

added blessed as peer dependency


[MISC] missing references in gettingstarted.rst

References for the errors page and listeners page lead to missing files inside the Handling my errors section in https://github.com/goverfl0w/discord-interactions/blob/stable/docs/gettingstarted.rst

It works on actual RTD site for me. https://discord-interactions.readthedocs.io/en/latest/gettingstarted.html

i viewed the file on github so i suppose this is isn't an error right?

yea the docs is meant to be compiled and viewed on readthedocs site, not on github.

Ok, Thank you for the help.


Add ascli as a new simplified tools command

Signed-off-by: Ahmed Hussein (amahussein<EMAIL_ADDRESS>Fixes #425, Fixes #436

adds a new ascli qualification cmd
platform default value is onPrem
the platform is not yet detected from the eventlogs path
adds a new ascli profiling cmd
platform default value is onPrem
todo: for profiling detect whether cluster is a CSP properties or worker's info
custome file

The documentation will be created when the new CLI is fully implemented because we will need to restructure the documentation so that we highlight that there is still option to use the spark_rapids_user_tools which allows more flexibility in passing the jar file, and the arguments passed to the rapids tool.
examples usage
run ascli help
$> ascli - --help

NAME
    ascli - CLI to run tools associated with RAPIDS Accelerator for Apache Spark plugin.

SYNOPSIS
    ascli - COMMAND

DESCRIPTION
    A wrapper script to run RAPIDS Accelerator tools (Qualification, Profiling, and Bootstrap)
    locally on the dev machine.

COMMANDS
    COMMAND is one of the following:

     profiling
       Provides a wrapper to simplify the execution of RAPIDS Profiling tool.

     qualification
       Provides a wrapper to simplify the execution of RAPIDS Qualification tool.

run dataproc qualification
ascli qualification --platform dataproc \
   --cluster $CPU_CLUSTER_PROPS \
   --output_folder $LOCAL_OUTPUT_DIR \
   --eventlogs $REMOTE_EVENT_LOGS

qualification help cmd
$> ascli qualification --help

NAME
  ascli qualification - Provides a wrapper to simplify the execution of
  RAPIDS Qualification tool.

SYNOPSIS
  ascli qualification <flags>

DESCRIPTION
  The Qualification tool analyzes Spark events generated from CPU based Spark
  applications to help quantify the expected acceleration and costs savings
  of migrating a Spark application or query to GPU.
  The wrapper downloads dependencies and executes the analysis on the local
  dev machine.

FLAGS
  --eventlogs=EVENTLOGS
    Type: Optional[str]
    Default: None
    Event log filenames or gs storage directories containing event logs
    (comma separated).

    Skipping this argument requires that the cluster argument points to a valid
    cluster name on the CSP.
  --cluster=CLUSTER
    Type: Optional[str]
    Default: None
    Name of cluster or path to cluster-properties. Note that using a "file path"
    requires the `platform` argument.
  --platform=PLATFORM
    Type: str
    Default: 'ONPREM'
    defines one of the following "onprem", "emr", "dataproc", "databricks".
  --output_folder=OUTPUT_FOLDER
    Type: Optional[str]
    Default: None
    path to store the output
  --target_platform=TARGET_PLATFORM
    Type: Optional[str]
    Default: None
    Cost savings and speedup recommendation for comparable cluster in `target_platform`
    based on on-premises cluster configuration. Requires cluster.
  --filter_apps=FILTER_APPS
    Type: str
    Default: 'SAVINGS'
    filtering criteria of the applications listed in the final STDOUT table is one of
    the following (NONE, SPEEDUPS, savings). Requires "Cluster".

    Note that this filter does not affect the CSV report. "NONE" means no filter applied.
    "SPEEDUPS" lists all the apps that are either 'Recommended', or 'Strongly Recommended'
    based on speedups. "SAVINGS" lists all the apps that have positive estimated GPU savings
    except for the apps that are "Not Applicable"
  --gpu_cluster_recommendation=GPU_CLUSTER_RECOMMENDATION
    Type: str
    Default: 'MATCH'
    The type of GPU cluster recommendation to generate. Requires "Cluster".
    It accepts one of the following:

    "MATCH": keep GPU cluster same number of nodes as CPU cluster;
    "CLUSTER": recommend optimal GPU cluster by cost for entire cluster;
    "JOB": recommend optimal GPU cluster by cost per job

run dataproc profiling
ascli profiling --platform dataproc \
   --cluster $CPU_CLUSTER_PROPS \
   --output_folder $LOCAL_OUTPUT_DIR \
   --eventlogs $REMOTE_EVENT_LOGS

run profiling help
NAME
  ascli profiling - Provides a wrapper to simplify the execution of
  RAPIDS Profiling tool.

SYNOPSIS
  ascli profiling <flags>

DESCRIPTION
  The Profiling tool analyzes GPU event logs and generates information
  which can be used for debugging and profiling Apache Spark applications.

FLAGS
  --eventlogs=EVENTLOGS
    Type: Optional[str]
    Default: None
    Event log filenames or CSP storage directories containing event logs
    (comma separated).

    Skipping this argument requires that the cluster argument points to a
    valid cluster name on the CSP.
  --cluster=CLUSTER
    Type: Optional[str]
    Default: None
    The cluster on which the Apache Spark applications were executed. It can
    either be a CSP-cluster name or a path to the cluster/worker's info
    properties file (json format).
  --platform=PLATFORM
    Type: str
    Default: 'ONPREM'
    defines one of the following "onprem", "emr", "dataproc", "databricks-aws",
    and "databricks-azure".
  --output_folder=OUTPUT_FOLDER
    Type: Optional[str]
    Default: None
    path to store the output

Why does ascli help require additional dash?  I would prefer to just do ascli --help rather than ascli - --help.


HoloLens 2 viewfinder appearance is unreliable only in OpenXR apps

Windows version: 22621.1376.arm64fre.ni_release_svc_sydney_rel_prod.240424-0600
OpenXR Runtime: 113.2405.31001.0 (preview enabled)
To reproduce:

Restart the HoloLens 2 (every time!)
Start an OpenXR application (e.g., BasicXrApp, OpenXR Tools Sample, etc)
Start MRC (e.g., via the "Start Recording" HL2 command or similar)

The missing viewfinder occurs after a fresh boot and the mixed reality camera has not been initiated outside of an OpenXR application. However, if you start the camera from outside the application and optionally stop it, the viewfinder will appear as normal in the OpenXR application, no matter how many times you start and stop MRC.
For more context. In the comments, @maluoi provides a possible cause:

I wonder if this is maybe a composition layer order issue? HL2's viewfinder composition layer sorts itself underneath [the OpenXR app's] projection layer in some cases. If that is what's happening, then this might be very tricky to solve on our end. We don't really have control over ordering of composition layers outside of our own.

Any updates on this, this bug seems to be quite consistent?


npm:specifier support

we need deno_emit to support npm: specifier also

This is a duplicate of  #73.


Please help to use it, i'm a beginner :(

i'm a beginner and i dont understand how to use it, i read wiki but i cant understand it, i very grateful for some one can explain it clearly or have video tutorial to use it, it just not for me, but it for me and many people cant understand how to use it

Did you even check youtube?

no, i search it but cant found it, you have tutorial in youtube ? can give link ? please

https://youtu.be/KiL579gt310


Fbar, the kernel's barriers limitation

Hi,
I would like to get more details about the fbar information reported by rocprof.
What does it mean.
How can it be interpreted.
Is it related to :
Max fbarriers/Workgrp:   32
Which is given by rocminfo.
Thanks

@etiennemlb Apologies for the lack of response. Do you still need assistance with this ticket? If not, please close the ticket. Thanks!

Yes I would like more/extensive documentation on the counter the AMD GPUs provide. Do you have a document I could rely on ?
Thanks!

Hi @etiennemlb, thanks for you patience.
Here is a link to the documentation for rocprofv1. However, note that fbar is a deprecated metric from rocprofv1.
Moreover, rocprofv1 is no longer under development and there is a new rocprofv3 released in ROCm 6.2 as a beta, which is built on top of the new rocprofiler-sdk. I strongly suggest using rocprofv3 since it is very close to having feature parity, has a lower overhead than v1 and v2, and is significantly better tested.
Here is a link to the documentation for rocprofv3 (See the "Using rocprofv3" section). You can also refer to this link for information on relevant performance counters and metrics or try "rocprofv3 --list-metrics" for a list of basic HW counters on your system.
Please let me know if need assistance with anything else!

Great! And if you encounter any issues with rocprofv3, you can create another ticket under the rocprofiler-sdk repository.


Implement new configuration system including JSON-Schema validation

Implement a better configuration system that avoids constant re-reading and re-writing of the config files by content caching and implicitly checks the well-formedness of the files using JSON-Schema.
Supersedes #365 and #190

Done with #416


We need 100% clarification on what traffic will go through the proxy

Having read the majority of issues/posts in this repo, I understand the following to be true:

All first party traffic will not go through the proxy
a. i.e. If you visit www.companyurl.com, all of the below example files would receive the originator IP address and not a proxy IP
i. www.companyurl.com/image1.png
ii. www.companyurl.com/file.js
iii. www.companyurl.com/styles.css
All third party traffic that is not listed by Google as a cross-site tracking script/tool will not go through the proxy
a. i.e. If www.companyurl.com loads a JS file from a third party (www.thirdparty.com/calc.js) that is not listed by Google as a cross-site tracking domain, the third party (www.thirdparty.com) will receive the originator IP address and not a proxy IP
Only domains listed by Google will go through the proxy
a. There is no plan to send everything through the proxy

Please could you confirm or correct each of the points above so that we are 100% clear.
Please could you also answer the following questions:

Is it just cross-site tracking domains that Google plans to send through the proxy?
Do Google plan to send other website analytics domains through the proxy that don't partake in cross-site tracking?
Do Google have a proposed list of domains that will be added to the proxy list that you can share with us?

I really appreciate you taking the time to clarify my understanding and answer my questions.
Thank you

@humbertoby8212 that is my understanding, but would be nice for someone from Google to definitively answer this. From what I can see, there is a fair amount of fear/uncertainty/doubt because of the lack of clarity on this. See also #13.

Hi @humbertoby8212
You are correct on the first three points (see below for more details).
Chrome’s IP Protection uses a list-based approach to identify which third-party traffic goes through the proxies. Origins that are on the list but are accessed in a first-party context will not be proxied through this service for those connections.
For example, if an analytics company is on the list of domains and a user navigates directly to the site, that site will still be able to observe the user’s IP address instead of the proxied IP address. However, if that domain on the list makes a network request in a third-party context, the connection will be proxied and the user's original IP address will not be visible to the site.
Our ultimate goal is to prevent cross-site tracking of users across the web. We are working through some details before sharing more information about which third-party domains we plan to focus on initially.

Can a bad actor exploit this mechanism to access free proxies? For example, could they trick Chrome into treating every request as a third party? I can see how free scan/DoS tools could route their traffic through Google’s proxy servers, which would put a strain on site owners by making it harder to block attackers, as they’d have easy and standardized proxy routes to targeted sites.
Although, even script kiddies can download a VPN or use residential proxies, making it difficult to block the attacking IP.
You’re right, but based on our analysis, most attackers don’t go that route. Simply blocking their IPs temporarily often ends their efforts, and they usually give up. From a security standpoint, even mitigating those script kiddies helps reduce the risk of accidental vulnerability discovery, which is a significant benefit.


Cleanup some IDExxxx and CAxxxx violations

For some reason the build fails locally when I add FxCopAnalyzers package to the csproj, Pushing to see what's the CI opinion about that 😄

The build fails when I add FxCopAnalyzers package to the csproj. I've no idea what's causing that.

This gone bigger than I thought :smile:
Feel free if you want to only address part of the changes or if you don't want to accept stylistic changes at all.
Probably the only important change is the missing dispose detected by roslyn-analyzers.

I'm going to close for now, it's better to split this in two or more PRs (code-quality & code-style should be separate for easier review)


PdoStatement fix (Issue #1225) broke MSSQL PDOStatement::fetch()

Issue: https://github.com/propelorm/Propel2/issues/1225 (Jun 2016)
Commit: https://github.com/propelorm/Propel2/commit/b1f82623ea1b61c9b4246aa7cf9be4196a9652a0 (Nov 2016)
PR: https://github.com/propelorm/Propel2/pull/1295 (Nov 2016)
I used to be able to execute the following but the commit above has removed my ability to retrieve @PARAMINT from the SELECT statement:
DECLARE @PARAMINT INT
DECLARE @PARAMVAR VARCHAR(50)

EXECUTE MyStoredProc
   @PARAMVAR = :PARAMVAR,
   @PARAMINT = @PARAMINT OUTPUT
SELECT @PARAMINT PARAMINT

This is my methodology:
// WHERE $query is defined above
$stmt = $dbConn->prepare($query);
$stmt->bindValue(':PARAMVAR', 'test-value', 'VARCHAR');
$stmt->execute();

// TO retrieve the PARAMINT parameter from the SELECT statement
do {
    $result = $stmt->fetch(\PDO::FETCH_OBJ);
    // THE SELECT STATEMENT USED TO RETURN AN OBJECT HERE
    if (is_object($result)) {
        return result;
    }
} while ($stmt->nextRowset());

Notes:

This issue only happens when the SELECT statement is outside the Stored Proc
Results are returned as per usual if the SELECT  statement is in the Stored Proc

Without having to rewrite my stored procs is there any workarounds for this issue?

OS: CentOS 7.4
PHP: 5.6
DB: MS SQL
Framework: Symfony 2
ORM: Propel

Could You find the cause of the problem by debugging Your code and searching for the change that broke Your code? I don't have MSSQL so I'm unable to reproduce this.

Thanks for your response. I will look more into.

Is this still relevant or can it be closed?

I'm no longer working on MSSQL DBs so I can't verify whether this is still an issue.
For that reason, maybe it can be closed.


Error handler: Forward event bus error details to console

Currently, we just print the error object generated by the event bus. Additionally we need to dump the root cause as well as the event that triggered the problem.

Implemented on master (v2.0.0).
Upgrade Information
When a widget throws an exception from an event bus subscription handler, that exception as well as the event that provoked it will now be logged to the browser console.


New MLBlocks API

A new API for MLBlocks will be implemented.
These are the most relevant changes:

Primitives JSONs and Python code has been moved to a different repository, called MLPrimitives
Optional usage of multiple JSON primitive folders.
JSON format has been changed to allow more flexibility and features:

input and output arguments, as well as argument types, can be specified for each method
both classes and function as primitives are supported
multitype and conditional hyperparameters fully supported
data modalities and primitive classifiers introduced
metadata such as documentation, description and author fields added

Parsers are removed, and now the MLBlock class is responsible for loading and reading the JSON primitive.
Multiple blocks of the same primitive are supported within the same pipeline.
Arbitrary inputs and outputs for both pipelines and blocks are allowed.
Shared variables during pipeline execution, usable by multiple blocks.

Closed via #44


Enable bootstrap for single host

This PR adds some things that have helped with build test environments with less post deploy manual config:

add arbitrary bootstrap arguments
add generic post bootstrap config commands
parameterise distribution release (required on Jammy or future newish releases)
make ssh_user specification optional
add generic post bootstrap and osd commands

Happy to further work on this if things are missing.
Cheers,
Piers.
Signed-off-by: Piers Harding<EMAIL_ADDRESS>

Hi - I resolved a merge conflict with upstream, but it would be good if you consider whether it still does what you intended - thanks.

Thanks @piersharding


[PE-D][Tester B] Unable to add student with valid input (Telegram handle: @11111111)

Perhaps the telegram handle follow (@11111111) the required format which is start with '@', no spaces in between the value and any values. But the application does not allow me to add the student with telegram handle @11111111.

Labels: severity.High type.FunctionalityBug
original: NgChunMan/ped#9

Duplicate error


[MINOR] Disable doris module since it is not ready for shipping

What changes were proposed in this pull request?
Don't package the Doris module into released package, since it is not ready for shipping. We could enable this when it is functional-ready.
Why are the changes needed?
To avoid ship with incomplete feature.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
No.

@zhoukangcn would you please help to check this? Currently I disabled this module to avoid packaging into the release package. If it could catch the 0.5.0 deadline (Apr. 14th), we can enable this.

@jerryshao I will try my best to catch the 0.5.0 deadline.  The code development is done, I need to split and submit the PR.


OCPBUGS-23832: CVE-2023-47108: bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp to v0.46

Bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp to fix CVE-2023-47108. cc @openshift/storage

/jira refresh

/lgtm

/retest

/cherry-pick release-4.15


gccgo: crash on implicit duplicate array/slice index

What version of Go are you using (go version)?
$ gccgo --version
gccgo (Debian 8.2.0-7) 8.2.0

Does this issue reproduce with the latest release?
maybe
What did you do?
package main

var f = [...]bool{3: false, 2: true, true} // the implicit index for the third element is 3

func main() {
}

What did you expect to see?
report error without crashing
What did you see instead?
crash
$ go run -compiler=gccgo aa.go
# command-line-arguments
go1: internal compiler error: in output_constructor_regular_field, at varasm.c:5142
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-8/README.Bugs> for instructions.

Playground reports the error: prog.go:3:38: duplicate index in array literal: 3
https://play.golang.org/p/_fVK5DNyb2G


Add 10Xv2 support to kallisto workflow

This PR adds a bit more flexibility to the kallisto workflow, starting work on https://github.com/AlexsLemonade/alsf-scpca/issues/52.  The main thing is that it now supports 10Xv2 and 10Xv3.1. It does this by now having a map (dictionary) of the 10X barcode files that can be checked when correcting cell barcodes as well as filtering to technology types that are in that list.
I checked that the workflow ran as expected with SCPCR000119 (v3.1) & SCPCR000120 (v2), though I have not looked at the actual count results.
In the process of  testing, I learned that bustools count will segfault if given the results from a bustools correct step where the barcode file is not found. You'd think bustools correct or the following bustools sort would fail, but they don't.
I also did a bit of cleanup, removing some unnecessary publish steps. I am still publishing the full bus directory, but only once now.

You said in your comments that you changed it to only publish the bus directory once but it appears that the publishDir is still in every process? Did you still mean for that to be the case?

No, I meant to take those out. I had an accidental loss of some edits at one point, and I guess I forgot to put those deletions back in.  Removing those for later.

Also the bustools_whitelist  process still exists but it's never used. Is this because of what you were referring to in terms of causing a segfault if it can't find the barcodes file? It looks like it should always be using one of the provided 10x files to me so theoretically we could remove that process. I tested it with removing that process with SCPCR000004 and it worked so was wondering if there is another reason why we needed it?

That was there when I thought we might want it to act a bit more like alevin and generate its own barcode list from the data. But this was not recommended, so I took out that step from the workflow, leaving the process definition in case we wanted to add it back at some point. We could remove it, but the cost is so small to leave it...

That was there when I thought we might want it to act a bit more like alevin and generate its own barcode list from the data. But this was not recommended, so I took out that step from the workflow, leaving the process definition in case we wanted to add it back at some point. We could remove it, but the cost is so small to leave it...

Understandable to leave it then. I just wanted to make sure I understood why it was there, since it wasn't necessary for the workflow to actually run, but perhaps maybe useful in the future.


Inference Upsample

For a resolution input of (800, 320), calculate the HeatMap on (800, 320) at inference time, and test the performance at the same size? Does performance degrade if points are directly resized to their original size? I didn't see the relevant upsampling code, I would like to ask where it is, thank you.

We resize the coordinates of the points by multiplying them by a scale factor. The code can be found here


fix: Text not itterable error when fast typing

What kind of change does this PR introduce?

Did you add tests for your changes?
If relevant, did you update the documentation?
Summary

Does this PR introduce a breaking change?

Other information

Bit tired rn, just woke up, but I can describe it. If you are typing in a text input after focusing it, sometimes there will be no data and so the handler inside the library expects an [] but gets undefined, the = [] defaults the value to an empty array. After testing it seemed to ahve fix any issues.
sorry if this is unprofessional, xD im jsut som 15 yr old

I see. It looks like you should fix it on the level where the data gets undefined/empty. Also, = [] is not a good default value for strings, it should be = ''. Typing an empty string makes no sense so that input should be checked before calling Puppeteer.


Proper Default Handling for HasKind/kind

Part 1 states that HasKind/kind has the default value Instance:

However, the current behaviour is that, for instance, a Submodel with no kind element returns null if executing getKind().
Proposal for discussion (alternative#1):
Let's change the functionality for getKind() to something like if (this.kind != null) { return this.kind } else { return ModelingKind.Intance }
pro:

if the deserialised Submodel has no kind attribute, so has its serialised version (would be different if we initialise it with ModelingKind.Intance)
no null value returned as the actual value is not null

con:

customisation of the generator only for HasKind necessary

Alternative#2:
Initialise with ModelingKind.Intance
con:

customisation of the generator only for HasKind necessary
deserialisation-serialization cycle "creates" the hasKind attribute

Alternative#3:
leave it for the business logic to handle kind = null
pro:

no custom adjustments in the generator

con:

what if the business application developer is not aware of the default value?

Are you sure that with Alternative#1 serialization will not serialize the property? AFAIK e.g. Jackson Framework uses getters to access the object properties and therefore would probably serialize this as kind = Instance.
Alternative#4 would be to initialize it as in Alterantive#2 but only serialize it if it differs from the default value.
If I remember correctly, default values are defined for multiple elements, so there might be benefit in adopting a generic pattern to deal with default values, define them in the underlying ontology and handle them in the generator.

I totally agree with that. However, I want to point out that keeping the exact document in all details upon a deserialize-serialize circle would require substantial changes/extensions to the model as it requires to track for each property (or at least the ones with default value) whether it was present in the original file and the using this information upon serialization.
To keep things simple I would propose to either serialize properties with default value only when the value is different from the default value (to keep serialized document as short as possible) or to always serialize them.
Personally, I would prefer the first approach, but would also be ok with the latter.


Update needed for m4 to 1.4.19

The current m4 currently used in version 1.4.18 is not compatible with glibc 2.34; this is fixed in 1.4.19.
(Context: we consider using these m4 bazel rules in Verible
https://github.com/chipsalliance/verible/issues/957 - but similar to jmillikin/rules_m4 that we're using now, the library is out of date)

Is this something @tgorochowik should have a look ?


Error generating report docker

Hey!
This plugin use to work fine for me, but since upgrading to 11 grafana this doesn't seem to work. I did try downgrading Grafana to 10.4.11 but I had the same issue.
I'm not sure if there are requirements I need to add to a dashboard but I just added a simple dashboard from a zabbix datasource the error i get is below
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T08:55:54.460138565Z level=warn msg="failed to get panels from browser data" endpoint=callResource error="browser: no dashboard data found" pluginId=mahendrapaipuri-dashboardreporter-app
logger=context userId=2 orgId=1 uname=x  t=2024-11-19T09:00:15.563236281Z level=info msg="Request Completed" method=GET path=/api/live/ws status=400 remote_addr=x time_ms=5 duration=5.716568ms size=12 referer= handler=/api/live/ws status_source=server
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T09:00:18.986474273Z level=info msg="generate report using config: Theme: dark; Orientation: portrait; Layout: simple; Dashboard Mode: full; Time Zone: Europe/London; Encoded Logo: ; Max Renderer Workers: 2; Max Browser Workers: 2; Remote Chrome Addr: http://renderer:8081/render; App URL: https://url-of-grafana; TLS Skip verify: true; Included Panel IDs: all; Excluded Panel IDs: none Included Data for Panel IDs: none" endpoint=callResource pluginId=mahendrapaipuri-dashboardreporter-app
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T09:00:18.986586577Z level=info msg="generate report using remote chrome" endpoint=callResource pluginId=mahendrapaipuri-dashboardreporter-app
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T09:00:18.992007838Z level=error msg="got error from clear browser cookies" endpoint=callResource error="failed to modify wsURL: invalid character '<' looking for beginning of value" pluginId=mahendrapaipuri-dashboardreporter-app
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T09:00:19.073326857Z level=warn msg="error fetching dashboard from browser, falling back to API" endpoint=callResource error="error fetching dashboard from browser: NavigateAndWaitFor: error enable lifecycle events: failed to modify wsURL: invalid character '<' looking for beginning of value" pluginId=mahendrapaipuri-dashboardreporter-app
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T09:00:19.073376558Z level=warn msg="failed to get panels from browser data" endpoint=callResource error="browser: no dashboard data found" pluginId=mahendrapaipuri-dashboardreporter-app
logger=rendering renderer=http t=2024-11-19T09:00:19.116517442Z level=info msg=Rendering path="d-solo/fe3xelrgs3z0gc/_?from=now-1h&height=500&panelId=1&theme=dark&to=now&width=1000"
logger=rendering renderer=http t=2024-11-19T09:00:19.128962312Z level=debug msg="calling remote rendering service" url="http://renderer:8081/render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=WY6fVUYsAN9VPYmHqabzmhU98k8OUh6Y&timeout=60&timezone=&url=http%3A%2F%2Fgrafana%3A3000%2Fd-solo%2Ffe3xelrgs3z0gc%2F_%3Ffrom%3Dnow-1h%26height%3D500%26panelId%3D1%26theme%3Ddark%26to%3Dnow%26width%3D1000%26render%3D1&width=1000"
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T09:00:22.216242052Z level=error msg="got error from clear browser cookies" endpoint=callResource error="failed to modify wsURL: invalid character '<' looking for beginning of value" pluginId=mahendrapaipuri-dashboardreporter-app
logger=context userId=2 orgId=1 uname=x t=2024-11-19T09:00:22.216684765Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=x time_ms=3394 duration=3.394113068s size=24 referer= handler=/api/plugins/:pluginId/resources/* status_source=downstream
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T09:00:22.216296254Z level=error msg="error generating report" endpoint=callResource err="error rendering PDF for dashboard New dashboard: error rendering PDF: error rendering PDF: failed to modify wsURL: invalid character '<' looking for beginning of value" pluginId=mahendrapaipuri-dashboardreporter-app

To confirm i have external service accounts enabled, the renderer is a seperate container but i set the URL in the plugin settings to point at the renderer. The container is in swarm mode.

Hey @RyanMesser
Thanks for the logs. Which version of plugin are you using?
Can you try to generate report for this very simple dashboard and post the logs (in DEBUG mode please)? I see you are using remote chrome as well? Which container image are you using for remote chrome instance?
Cheers

Hey @RyanMesser
Thanks for the logs. Which version of plugin are you using?
Can you try to generate report for this very simple dashboard and post the logs (in DEBUG mode please)? I see you are using remote chrome as well? Which container image are you using for remote chrome instance?
Cheers

Hey,
I'm using 1.7.0.
I willl post the logs tomorrow but I quickly tried just now and I get the same issue with the simple dashboard.
Yes i'm using the latest grafana-image-renderer but I also tried an earlier version with no avail. The logs for that are below this is in debug mode
DBG DBG DBG DBG Render request received | DBG DBG DBG DBG Render request received | url=http://grafana:3000/d-solo/e472bbf0-140c-4852-a74b-1a4c32202659/_?from=now-5m&height=500&panelId=25&theme=dark&to=now&var-testvar0=%24__all&var-testvar1=foo&var-testvar2=1&width=1000&render=1
DBG DBG DBG DBG Setting viewport for page | DBG DBG DBG DBG Setting viewport for page | deviceScaleFactor=1 DBG DBG DBG DBG Setting viewport for page | DBG DBG DBG DBG Setting viewport for page | deviceScaleFactor=1 height=500 DBG DBG DBG DBG Setting viewport for page | DBG DBG DBG DBG Setting viewport for page | deviceScaleFactor=1 DBG DBG DBG DBG Setting viewport for page | DBG DBG DBG DBG Setting viewport for page | deviceScaleFactor=1 height=500 width=1000
DBG DBG DBG DBG Setting cookie for page | DBG DBG DBG DBG Setting cookie for page | domain=grafana DBG DBG DBG DBG Setting cookie for page | DBG DBG DBG DBG Setting cookie for page | domain=grafana renderKey=AhF4hvhsAZH5FhqXXaCenP49fVxWrgfv
DBG DBG DBG DBG Moving mouse on page | DBG DBG DBG DBG Moving mouse on page | x=1000 DBG DBG DBG DBG Moving mouse on page | DBG DBG DBG DBG Moving mouse on page | x=1000 y=500
DBG DBG DBG DBG Navigating and waiting for all network requests to finish | DBG DBG DBG DBG Navigating and waiting for all network requests to finish | url=http://grafana:3000/d-solo/e472bbf0-140c-4852-a74b-1a4c32202659/_?from=now-5m&height=500&panelId=25&theme=dark&to=now&var-testvar0=%24__all&var-testvar1=foo&var-testvar2=1&width=1000&render=1
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/d-solo/e472bbf0-140c-4852-a74b-1a4c32202659/_?from=now-5m&height=500&panelId=25&theme=dark&to=now&var-testvar0=%24__all&var-testvar1=foo&var-testvar2=1&width=1000&render=1
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/grafana.dark.722d809dba5a31f57d49.css
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/runtime.5d3ab526a3394c8ca029.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/6029.0549a3fcb50e73c4b256.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/4239.c2eca2b0222ddf40980b.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/3379.020d2edcd8eb1d9f6d9a.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/3719.c065b2e146c4c8347d51.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/8078.a4260b06f1454becf5c9.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/322.9e448e90c86f8dd2907e.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/app.a986507559398be8774a.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/grafana.dark.722d809dba5a31f57d49.css
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/d-solo/e472bbf0-140c-4852-a74b-1a4c32202659/_?from=now-5m&height=500&panelId=25&theme=dark&to=now&var-testvar0=%24__all&var-testvar1=foo&var-testvar2=1&width=1000&render=1
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/img/grafana_icon.svg
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/runtime.5d3ab526a3394c8ca029.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/img/grafana_icon.svg
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/6029.0549a3fcb50e73c4b256.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/3719.c065b2e146c4c8347d51.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/8078.a4260b06f1454becf5c9.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/4239.c2eca2b0222ddf40980b.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/app.a986507559398be8774a.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/322.9e448e90c86f8dd2907e.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/3379.020d2edcd8eb1d9f6d9a.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/9521.fccdbf0802dd2985e2fc.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/9521.fccdbf0802dd2985e2fc.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/9569.817db115a7df4172282b.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/SoloPanelPage.e211f5ee1b174e04be0b.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/fonts/inter/Inter-Regular.woff2
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/fonts/inter/Inter-Regular.woff2
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/img/fav32.png
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/9569.817db115a7df4172282b.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/SoloPanelPage.e211f5ee1b174e04be0b.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/img/fav32.png
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/api/dashboards/uid/e472bbf0-140c-4852-a74b-1a4c32202659
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/img/icons/unicons/spinner.svg
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/img/icons/unicons/spinner.svg
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/api/dashboards/uid/e472bbf0-140c-4852-a74b-1a4c32202659
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/grafanaPlugin.abd290ae5943e08a7661.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/4007.cfd25ba5558e3595af64.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/3397.ddbb0c6f910bdfcd328d.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/94.ebca783641d5754a27ec.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/6965.0c5239d653c3175969b3.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/build/timeseriesPanel.ac61c60b21097a7e8bf0.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/grafanaPlugin.abd290ae5943e08a7661.js
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/api/annotations?from=1732053862707&to=1732054162707&limit=100&matchAny=false&dashboardUID=e472bbf0-140c-4852-a74b-1a4c32202659
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/4007.cfd25ba5558e3595af64.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/3397.ddbb0c6f910bdfcd328d.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/6965.0c5239d653c3175969b3.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/timeseriesPanel.ac61c60b21097a7e8bf0.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/build/94.ebca783641d5754a27ec.js
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/api/annotations?from=1732053862707&to=1732054162707&limit=100&matchAny=false&dashboardUID=e472bbf0-140c-4852-a74b-1a4c32202659
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=POST DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=POST url=http://grafana:3000/api/ds/query?ds_type=datasource&requestId=SQR100
DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET DBG DBG DBG DBG Browser request | DBG DBG DBG DBG Browser request | method=GET url=http://grafana:3000/public/fonts/inter/Inter-Medium.woff2
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=POST DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=POST url=http://grafana:3000/api/ds/query?ds_type=datasource&requestId=SQR100
DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET DBG DBG DBG DBG Browser request finished | DBG DBG DBG DBG Browser request finished | method=GET url=http://grafana:3000/public/fonts/inter/Inter-Medium.woff2
ERR ERR ERR ERR Browser console error | ERR ERR ERR ERR Browser console error | column=6214 ERR ERR ERR ERR Browser console error | ERR ERR ERR ERR Browser console error | column=6214 line=51 ERR ERR ERR ERR Browser console error | ERR ERR ERR ERR Browser console error | column=6214 ERR ERR ERR ERR Browser console error | ERR ERR ERR ERR Browser console error | column=6214 line=51 msg=Moment Timezone found Etc/Unknown from the Intl api, but did not have that data loaded. ERR ERR ERR ERR Browser console error | ERR ERR ERR ERR Browser console error | column=6214 ERR ERR ERR ERR Browser console error | ERR ERR ERR ERR Browser console error | column=6214 line=51 ERR ERR ERR ERR Browser console error | ERR ERR ERR ERR Browser console error | column=6214 ERR ERR ERR ERR Browser console error | ERR ERR ERR ERR Browser console error | column=6214 line=51 msg=Moment Timezone found Etc/Unknown from the Intl api, but did not have that data loaded. url=http://grafana:3000/public/build/4239.c2eca2b0222ddf40980b.js
DBG DBG DBG DBG Waiting for dashboard/panel to load | DBG DBG DBG DBG Waiting for dashboard/panel to load | timeout=60s
DBG DBG DBG DBG Setting zoom level | DBG DBG DBG DBG Setting zoom level | zoomLevel=1
DBG DBG DBG DBG Taking screenshot | DBG DBG DBG DBG Taking screenshot | filePath=/tmp/c35add36.png
DBG DBG DBG DBG Deleting temporary file | DBG DBG DBG DBG Deleting temporary file | file=/tmp/c35add36.png
DBG DBG DBG DBG ::ffff:<IP_ADDRESS> - - [19/Nov/2024:22:09:23 +0000] "GET /render?deviceScaleFactor=1.000000&domain=grafana&encoding=png&height=500&renderKey=AhF4hvhsAZH5FhqXXaCenP49fVxWrgfv&timeout=60&timezone=&url=http%3A%2F%2Fgrafana%3A3000%2Fd-solo%2Fe472bbf0-140c-4852-a74b-1a4c32202659%2F_%3Ffrom%3Dnow-5m%26height%3D500%26panelId%3D25%26theme%3Ddark%26to%3Dnow%26var-testvar0%3D%2524__all%26var-testvar1%3Dfoo%26var-testvar2%3D1%26width%3D1000%26render%3D1&width=1000 HTTP/1.1" 200 37520 "-" "Grafana/11.3.0+security-01"

DBG DBG DBG DBG Connection closed | DBG DBG DBG DBG Connection closed | url=http://grafana:3000/d-solo/e472bbf0-140c-4852-a74b-1a4c32202659/_?from=now-5m&height=500&panelId=25&theme=dark&to=now&var-testvar0=%24__all&var-testvar1=foo&var-testvar2=1&width=1000&render=1
ERR ERR ERR ERR ::ffff:<IP_ADDRESS> - - [19/Nov/2024:22:09:23 +0000] "GET /json/version HTTP/1.1" 404 151 "-" "Go-http-client/1.1"

ERR ERR ERR ERR ::ffff:<IP_ADDRESS> - - [19/Nov/2024:22:09:23 +0000] "GET /json/version HTTP/1.1" 404 151 "-" "Go-http-client/1.1"

Hey there, Thanks again for the logs.
I was talking about the chromium container image and not grafana-image-renderer. The plugin needs chromium installed either in Grafana container image or run as a separate container. From the following log line:
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-11-19T09:00:18.986586577Z level=info msg="generate report using remote chrome" endpoint=callResource pluginId=mahendrapaipuri-dashboardreporter-app

I see you are using remote chromium instance. I would like to know which image are you using for that remote chrome.
If you can share a docker compose file of your environment, that would help me to reproduce your issue and find the cause.
Cheers!!

Will have to scrub but I can sure share.
Grafana Image renderer uses chromium browser so maybe this is my misunderstanding in that the plugin would utilise this? In the plugin config I pointed the remote chrome instance at http://renderer:8081 for example so maybe this is my mistake?
Appreciate the help!

@RyanMesser Glad that you figured it out.
Yes, the current plugin will use the chromium shipped by grafana-image-renderer when you install the grafana-image-renderer on Grafana server as plugin. Ad you are running grafana-image-renderer as a separate service, we the current reporter plugin do not have access to chromium binary shipped by grafana-image-renderer.
I remember trying to start the chromium of grafana-image-renderer with remote debugging protocol, i.e., setting env var RENDERING_ARGS=--no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--remote-debugging-port=9223 to renderer container. Then you will need to expose the port 9223 of renderer container and then using ws://renderer:9223 as remote chromium. However, I remember this didnt work as chromium of grafana-image-renderer never opened the port of remote debugging protocol. Probably this is due to standalone chromium shipped by grafana-image-renderer that potentially does not support all features. Maybe you can give it a go if something has changed since the last time I tried.

@mahendrapaipuri  yeah I worked that out :)
I will try that on my dev container and report back.

Ok I've added the chrome remote container and pointed it at that and now it works :) Think it was a bit of me misunderstanding the renderer container.
chrome: image: chromedp/headless-shell:latest shm_size: 2G init: true networks: - shared_network

How did you point grafana at the remote chrome container?

Ok I've added the chrome remote container and pointed it at that and now it works :) Think it was a bit of me misunderstanding the renderer container.
chrome: image: chromedp/headless-shell:latest shm_size: 2G init: true networks: - shared_network

How did you point grafana at the remote chrome container?

In Grafana go to to the plugin config section, click advanced settings and add the remote chrome instance here

Hmm. I shouldn't have to expose the port on the chrome container right? I have a very similar docker-compose file to @RyanMesser.
The errors i'm seeing in the log are "got error from clear browser cookies" and "failed to modify wsURL"

@Gearbox007 No, you dont have to expose any port as long as you have init: true in your remote chrome container spec.
What did you use as remote chrome URL in the plugin config? It should be ws://localhost:9222 as init: true ensures that chrome container is in same local network as Grafana server. Could you try with that please?

ws://localhost:9222 did it!
thank you!


maintainers: update name to anna

Motivation for this change
I have changed my GitHub username and have started using a different name. I am changing my entry in the maintainers list to match this.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Thanks, I've merged this for you.


Build MicroPython for multiple boards

Upgrades the CI to output builds for:

Pico (2MB)
Tiny 2040 (8MB)
Pico LiPo (4MB)
Pico LiPo (16MB)

Provides the 16MB Flash build requested in #272
I've added some significant caching both via ccache (for caching compiler output) and GitHub Actions cache.
MicroPython and the Pico SDK are cloned, prepped and cached per version- this cache should probably be used by other workflows to speed up these stages since the MicroPython version changes very rarely.
Installing ARM GCC is still a huge time sink- it needs to be done for each port from the list above and can take anywhere between 1 and 2 minutes, even hanging for much, much longer- this quickly multiplies up across all builds. Could we grab the zipped toolchain and cache it?

Managed to slash the ARM GCC install time by wget'ing the toolchain, extracting it and adding it to PATH. This is surprisingly easy to do as a drop-in replacement to grabbing it from apt and seems to take an approximately constant 30sec with no drawn-out hangs.


Add config/map/token endpoint to support Data Catalog

Description
This adds a new endpoint to the tiler service which returns an authorization token to be used to access rendering functionality of an azure maps instance. This is to facilitate removing an azure maps key from the Data Catalog application, and aquire an Entra ID based short lived token instead.
In order to support this endpoint, a number of changes are introduced:

The tiler chart is updated to annotate/label the service account and deployment to use azure workload identity
A new identity is created and used as the WLI above, and has RBAC permissions assigned to the appropriate maps role
A new azure maps resource is created for this environment, in order to assign the RBAC role (previously, a single instance supplied all environments)
The dev environment was modified to use azure cli identity in lieu of a managed identity
Tests were updated to toggle the use of a test the requires this identity; the CI environment does not have one

This was tested by deploying to the PCT environment.


[Task] Test Air gap with Rancher integration

We need to test the air gap support with Rancher integration, make sure not only Harvester works fine in the air gap environment by itself, but the user can also start a Rancher in the air gap environment and complete all the necessary tasks.

All known issues are resolved in the v1.0.0-rc1 release, please help to move on with more air-gap env testings, thanks.

According to task, we propose to begin the test coverage with the following scenarios
Environment Test Scenarios:

[ ] 1. Rancher and harvester are both under air gap environment within same subnet

[ ] 2. Rancher is under air gap environment while harvester have internet connection, they are located in same subnet

[ ] 3. Rancher have internet connection while harvester under air gap environment, located in different subnet

Validate point

Can import harvester from Rancher correctly
Can access downstream harvester cluster from Rancher dashboard
Can provision at least one node RKE1 cluster to harvester correctly with running status
Can explore provisioned RKE1 cluster nodes
RKE1 cluster VM created running correctly on harvester node
Can provision at least one node RKE2 cluster to harvester correctly with running status
Can explore provisioned RKE2 cluster nodes
RKE2 cluster VM created running correctly on harvester node

Test Scenario 2: PASS
Rancher is under air gap environment while harvester have internet connection, they are located in same subnet

[x] Can import downstream harvester from rancher

[x] Can access downstream harvester cluster from Rancher dashboard

[x] Can provision at least one node RKE1 cluster to harvester correctly with running status

[x] Can explore provisioned RKE1 cluster nodes

[x] RKE1 cluster VM created running correctly on harvester node

[x] Can provision at least one node RKE2 cluster to harvester correctly with running status

[x] Can explore provisioned RKE2 cluster nodes

[x] RKE2 cluster VM created running correctly on harvester node

Can import downstream harvester from rancher

Can access downstream harvester cluster from Rancher dashboard

Can provision at least one node RKE1 cluster to harvester correctly with running status

Can explore provisioned RKE1 cluster nodes

RKE1 cluster VM created running correctly on harvester node

Can provision at least one node RKE2 cluster to harvester correctly with running status

Can explore provisioned RKE2 cluster nodes

RKE2 cluster VM created running correctly on harvester node

Test Scenario 1: Ongoing
Rancher and harvester are both under air gap environment within same subnet
Test environment

Rancher: v2.6-head (77392a6)
Harvester: master-5d6f3db0-head

[x] Can import downstream harvester from rancher

[x] Can access downstream harvester cluster from Rancher dashboard

[ ] Failed to provision RKE2 cluster on harvester with failed calling webhook

@TachunLin can u please check the pod status via ssh login to the RKE2 node on the Harvester side, just want to double-check if that is caused by missing some RKE2 air-gap images, thanks.

@guangbochen, once clicking create button on rancher dashboard, it was hanging for seconds then directly pop out above error message of Internal error occurred: failed calling webhook "default.cluster.cluster.x-k8s.io"
Since the RKE2 node vm not yet be created on harvester side, it's not available to check the pod status inside the rke2 node.
I checked the docker logs on rancher side it has some syncing issue, I think can check the http proxy setting on rancher side to investigate the connectivity to internet through proxy.

@guangbochen, according to @gitlawr instruction, currently I can successfully provision RKE2 cluster on both harvester and rancher in air gap environment as Test scenario 1

[x] Can import downstream harvester from rancher

[x] Can access downstream harvester cluster from Rancher dashboard

[x] Can provision RKE2 cluster on harvester with failed calling webhook

To solve this error we need to fulfill the following criteria

Rancher docker run command requires http proxy arguments

sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 \
-e HTTP_PROXY="http://<IP_ADDRESS>:3128" \
-e HTTPS_PROXY="http://<IP_ADDRESS>:3128" \
-e NO_PROXY="localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/8,cattle-system.svc,<IP_ADDRESS>/16,.svc,.cluster.local,example.com" \
--privileged rancher/rancher:v2.6.3-rc6

Set the following in Advanced - user data in the create RKE2 cluster page

#cloud-config
apt:
  http_proxy: http://<IP_ADDRESS>:3128
  https_proxy: http://<IP_ADDRESS>:3128

In Cluster Configuration - Agent Environment Vars, add the following

HTTP_PROXY: http://<IP_ADDRESS>:3128
	HTTPS_PROXY: http://<IP_ADDRESS>:3128
	NO_PROXY: localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/8,cattle-system.svc,.svc,.cluster.local

@gitlawr @TachunLin can we please help to document this with an air-gap support page on http://<IP_ADDRESS>:8000/rancher/rancher-integration/#air-gap-support, thanks

Test Scenario 1: PASS
Rancher and harvester are both under air gap environment within same subnet
Test environment

Rancher: v2.6.3-rc6
Harvester: v.1.0.0-rc2

[x] Can import downstream harvester from rancher

[x] Can access downstream harvester cluster from Rancher dashboard

[x] Can provision at least one node RKE1 cluster to harvester correctly with running status

[x] Can explore provisioned RKE1 cluster nodes

[x] RKE1 cluster VM created running correctly on harvester node

[x] Can provision at least one node RKE2 cluster to harvester correctly with running status

[x] Can explore provisioned RKE2 cluster nodes

[x] RKE2 cluster VM created running correctly on harvester node

Steps to provision RKE1 cluster in both air gap environment

Clone ipxe example
Enable offline feature in settings.yaml
Use vagrant pxe example to build an one node harvester
Prepare a ubuntu VM and config virtual network harvester provided by pxe example
Run docker command to create rancher with http proxy arguments

sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 \
-e HTTP_PROXY="http://<IP_ADDRESS>:3128" \
-e HTTPS_PROXY="http://<IP_ADDRESS>:3128" \
-e NO_PROXY="localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/8,cattle-system.svc,<IP_ADDRESS>/16,.svc,.cluster.local,example.com" \
--privileged rancher/rancher:v2.6.3-rc6

Import harvester to rancher dashboard
Create cloud credential in rancher
Enable virtual network in harvester with harvester-mgmt
Create the virtual network vlan1 with id 1
Create RKE1 template
Input the following content in user data

#cloud-config
apt:
  http_proxy: http://<IP_ADDRESS>:3128
  https_proxy: http://<IP_ADDRESS>:3128
write_files:
- path: /etc/environment
  content: |
    HTTP_PROXY="http://<IP_ADDRESS>:3128"
    HTTPS_PROXY="http://<IP_ADDRESS>:3128"
  append: true

Expand Engine Options, add the following engine environment

HTTP_PROXY: http://<IP_ADDRESS>:3128
HTTPS_PROXY: http://<IP_ADDRESS>:3128
NO_PROXY: localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/8,cattle-system.svc,.svc,.cluster.local

Create a RKE1 cluster
Select harvester as cloud provider

Steps to provision RKE2 cluster in both air gap environment

Follow the same step 1 ~ 9 as RKE1 scenario
Create a RKE2 cluster page
Add the following content in User data under Advanced

#cloud-config
apt:
  http_proxy: http://<IP_ADDRESS>:3128
  https_proxy: http://<IP_ADDRESS>:3128

Open Agent Environment Vars
Add the following variable and value pair

HTTP_PROXY: http://<IP_ADDRESS>:3128
HTTPS_PROXY: http://<IP_ADDRESS>:3128
NO_PROXY: localhost,<IP_ADDRESS>,<IP_ADDRESS>,<IP_ADDRESS>/8,cattle-system.svc,.svc,.cluster.local

6. Click the create button