DANN implementation for image classification

Hello,
I am currently working on a domain adaptation project involving image datasets. I have successfully used the ADAPT package and especially DANN implementation in your tutorial, but I am unsure if the same techniques can be extended to image data.
I was wondering if you have examples of DANN implementation for computer vision tasks like multiclass classification, and if so what did you change in the parameters?
I am currently trying to apply your DANN implementation with a model trained on Mnist as source domain, to SVHN considered as target domain (or the opposite), would you happen to have tried it ?
Thank you for your time and for developing such a useful tool for domain adaptation. I look forward to your response and any guidance you can provide.
Best regards

Hi @zobapt,
Thank you for asking! Yes, Adapt can be used for image classification tasks! Here is an example for the adaptation from SVHN to MNIST that should run on Google Colab:
Here is the link to the colab: https://colab.research.google.com/drive/1P9LUA9UCGlvCCJs9dg244NdyKdIiHSCJ?usp=sharing
Here is the code:

# Install adapt and come back to Tensorflow 2.15 (Adapt does not work with TF 2.16)
!pip install adapt
!pip install tensorflow==2.15

# Import packages
import tensorflow as tf
from tensorflow.keras import layers
import tensorflow_datasets as tfds
from sklearn.manifold import TSNE
import numpy as np
import matplotlib.pyplot as plt

from adapt.feature_based import DANN

# Load datasets
source_train, source_test = tfds.load('svhn_cropped', split=['train', 'test'], shuffle_files=False)
target_train, target_test = tfds.load('mnist', split=['train', 'test'], shuffle_files=False)

# Preprocessing
# IMPORTANT: Deep learning methods from Adapt require tuples (images, labels) instead of dictionaries
# SVHN images are cropped from (32, 32) to (28, 28) and MNIST are duplicated along the last axis to get 3 channels
source_train = source_train.map(lambda x: ((tf.cast(x['image'][2:-2, 2:-2, :], "float64") - 127.5) / 127.5,
                                           tf.one_hot(x['label'], depth=10)))
source_test = source_test.map(lambda x: ((tf.cast(x['image'][2:-2, 2:-2, :], "float64") - 127.5) / 127.5,
                                         tf.one_hot(x['label'], depth=10)))

target_train = target_train.map(lambda x: ((tf.cast(tf.concat([x['image']]*3, axis=-1), "float64") - 127.5) / 127.5,
                                           tf.one_hot(x['label'], depth=10)))
target_test = target_test.map(lambda x: ((tf.cast(tf.concat([x['image']]*3, axis=-1), "float64") - 127.5) / 127.5,
                                         tf.one_hot(x['label'], depth=10)))

input_target_train = target_train.map(lambda x, y: x)        # Extract inputs for the target (target labels are not given during training)
input_source_train = source_train.map(lambda x, y: x)

# Define networks
def make_encoder_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(64, (5, 5), strides=(1, 1), padding='same',
                                     input_shape=[28, 28, 3]))
    model.add(layers.ReLU())
    model.add(layers.MaxPool2D((3, 3), strides=(2, 2)))

    model.add(layers.Conv2D(64, (5, 5), strides=(1, 1), padding='same'))
    model.add(layers.ReLU())
    model.add(layers.MaxPool2D((3, 3), strides=(2, 2)))

    model.add(layers.Conv2D(128, (5, 5), strides=(1, 1), padding='valid'))
    model.add(layers.ReLU())

    model.add(layers.Flatten())
    return model

def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(1024, input_shape=[512,]))
    model.add(layers.ReLU())

    model.add(layers.Dense(1024))
    model.add(layers.ReLU())

    model.add(layers.Dense(1))
    model.add(layers.Activation("sigmoid"))
    return model

def make_task_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(1024, input_shape=[512,]))
    model.add(layers.ReLU())

    model.add(layers.Dense(1024))
    model.add(layers.ReLU())

    model.add(layers.Dense(10))
    model.add(layers.Softmax())
    return model

# Instantiate DANN
encoder = make_encoder_model()
discriminator = make_discriminator_model()
task = make_task_model()

dann_model = DANN(encoder=encoder, discriminator=discriminator, task=task, lambda_=0.1, copy=False)
dann_model.compile(loss=tf.keras.losses.CategoricalCrossentropy(from_logits=False),
                   optimizer=tf.keras.optimizers.legacy.SGD(0.1),
                   metrics=["acc"])

# Train DANN
dann_model.fit(X=source_train, Xt=input_target_train, batch_size=128, epochs=10,
               shuffle=True, validation_data=target_test, buffer_size=1000)

Without adaptation (lambda_=0. in DANN)
Target test accuracy: ~0.53

With adaptation (lambda_=0.1 in DANN)
Target test accuracy: ~0.69

We are close to the results presented in the paper

I'm very grateful for your complete answer. I managed to reproduce it without any problems.
Thank you very much !


Add row direction support

this PR is for #1070 😊
all tests passing ✔
this adds row direction support to masonry (right to left cells)

@bardiarastin I'm thinking of instead of having a rowDirection prop, just have it as a opt-in boolean prop/.
For example:
<Masonry
  {...this.props}
  rtl
/>

@wuweiweiwu thanks for feedback, I had the same idea but this is from discussion in #1070

@bardiarastin I see. Looks good to me 👍

Hi 👋 Looks like this PR broke CI because it wasn't formatted with Prettyer (yarn prettier)
I fixed it (1e1746a) and pushed. Just an FYI.

thanks @bvaughn, I made another PR to add yarn prettier to contributing docs 😬

Woops. my bad. Do you think it is worth setting up CI for PRs? cc @bvaughn


CAAPH considers an image tag as number instead of string

/kind bug
[Before submitting an issue, have you checked the Troubleshooting Guide?]
What steps did you take and what happened:
[A clear and concise description of what the bug is.]
In a CI build with CAPZ 1.13, cloud-node-manager crashed because image tag is like this:
capzci.azurecr.io/azure-cloud-node-manager:%!s(float64=7.392922e+06)

I checked other logs, the actual image tag is 7392922.
I believe CAAPH didn't handle the tag as a string.
Actually, before CAAPH, the similar problem happens and we use helm --set-string
What did you expect to happen:
A number tag should be corrected parsed.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:

cluster-api-provider-azure version: 1.13
Kubernetes version: (use kubectl version):
OS (e.g. from /etc/os-release):

@Jont828 do you think this is a bug in CAAPH, or can we fix this by changing this:
https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/v1.13.1/templates/addons/cluster-api-helm/cloud-provider-azure-ci.yaml#L28
to this:
imageTag: "${IMAGE_TAG_CNM:-}"
cc @mboersma

@Jont828 I prefer to do it in code, so we don't need to update every template.

@jackfrancis @lzhecheng The valuesTemplate field is just a string that gets passed directly to Helm, and the env var substitution for IMAGE_TAG_CNM happens in CAPZ I believe. So I'm not sure if this is something we can address with CAAPH. Could we try Jack's suggestion of adding quotes first to see if it works?

@jackfrancis @lzhecheng The valuesTemplate field is just a string that gets passed directly to Helm, and the env var substitution for IMAGE_TAG_CNM happens in CAPZ I believe. So I'm not sure if this is something we can address with CAAPH. Could we try Jack's suggestion of adding quotes first to see if it works?

I'm ok with Jack's suggestion but I just wonder if there's a solution for all such fields.

I don't think there's an immediate solution that comes to mind since it seems like it's dependent on how Helm parses the values.

I checked cluster-api-addon-provider-helm code and have some ideas. Please correct me if I'm wrong.
Here, it uses ValueFiles for the whole spec. If IMAGE_TAG is put under StringValues then it should be correctly parsed.
https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/blob/378279b6c3d0153d1dd9d3496d2d9409d6118815/internal/helm_client.go#L260-L262
Or, if we can add quotes for numbers in valueLookUp in value_substitutions.go before returning expandedTemplate.
https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/blob/378279b6c3d0153d1dd9d3496d2d9409d6118815/internal/value_substitutions.go#L75
@Jont828 @jackfrancis what do you think?

I think adding a stringValues configuration interface to CAAPH if it doesn't already exists makes sense.
In the meantime: https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/4593

/priority backlog

@jackfrancis If your solution works, then I think we can open a PR to add that as documentation and close this issue then.

I think it works!


Docs: Add target option (Wheel Gestures Plugin)

add info about the newly added target option

Thank you for updating the docs @xiel ⭐.


module-info.java is not supported

Extension Name: vscode-gradle
Extension Version:v3.11.0
OS Version: windows10
VSCode version:1.66.1
Describe the bug
Could not build module project like module-info.java
To Reproduce

please read doc https://docs.gradle.org/current/samples/sample_java_modules_multi_project.html
download  https://docs.gradle.org/current/samples/zips/sample_java_modules_multi_project-groovy-dsl.zip
unzip sample_java_modules_multi_project-groovy-dsl.zip
open sample_java_modules_multi_project-groovy-dsl by vscode

Expected behavior
Build succeeded
Screenshots

Output from "Gradle for Java"

[info] [gradle-server] Server started, listening on 8451
[info] Gradle client connected to server
[info] Java Home: C:\Program Files\RedHat\java-17-openjdk-<IP_ADDRESS>.12-1
[info] JVM Args: --add-modules=jdk.jdeps,java.base,java.compiler,jdk.compiler,-XX:MaxMetaspaceSize=512m,-XX:+UseShenandoahGC,-XX:MinHeapFreeRatio=20,-XX:MaxHeapFreeRatio=40,-XX:GCTimeRatio=4,-XX:AdaptiveSizePolicyWeight=90,-XX:+HeapDumpOnOutOfMemoryError,-XX:ReservedCodeCacheSize=1g,-XX:SoftRefLRUPolicyMSPerMB=50,-XX:-OmitStackTraceInFastThrow,--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.base/java.lang=ALL-UNNAMED,--add-opens,java.base/java.lang.invoke=ALL-UNNAMED,--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens,java.base/java.nio.charset=ALL-UNNAMED,--add-opens,java.base/java.net=ALL-UNNAMED,--add-opens,java.base/java.util.concurrent.atomic=ALL-UNNAMED,-Xms1g,-Xmx6g,-Dfile.encoding=UTF-8,-Duser.country=CN,-Duser.language=zh,-Duser.variant
[info] Gradle User Home: D:\java\gradlehome
[info] Gradle Version: 7.4.2
[info] > Task :buildSrc:extractPluginRequests UP-TO-DATE
> Task :buildSrc:generatePluginAdapters UP-TO-DATE
> Task :buildSrc:compileJava UP-TO-DATE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:compileGroovyPlugins UP-TO-DATE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validatePlugins UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings

CONFIGURE SUCCESSFUL in 3s
9 actionable tasks: 9 up-to-date

Does the bug still exist if you disable all other extensions?
No
Additional context
Please help to confirm if it is a problem with the Gradle plugin

Please help to confirm if it is a problem with the Gradle plugin

I can reproduce this issue. It's a problem with vscode-java, the Gradle plugin only provides Gradle views and syntax check for *.gradle files.
It's an issue related to the multi-module project , I suggest you create an issue in vscode-java repository.

@CsCherrYY Thanks you!


Should micromatch package in orval/core be bumped to 4.0.8 due to Snyk vulnerability report?

What are the steps to reproduce this issue?

Run snyk test --severity-threshold=high on package after installing

What happens?
Receive error:
✗ Inefficient Regular Expression Complexity [High Severity][https://security.snyk.io/vuln/SNYK-JS-MICROMATCH-6838728] in<EMAIL_ADDRESS>introduced by<EMAIL_ADDRESS>><EMAIL_ADDRESS>><EMAIL_ADDRESS>><EMAIL_ADDRESS>and 1 other path(s)
This issue was fixed in versions: 4.0.8
What were you expecting to happen?
Snyk to be fine with all Orval dependencies
Any other comments?
Synk report is here: https://security.snyk.io/vuln/SNYK-JS-MICROMATCH-6838728
Does this affect us and should we bump micromatch in orval/core to 4.0.8?
What versions are you using?
npmPackages:
@tanstack/svelte-query: ^5.56.2 => 5.56.2
axios: ^1.7.7 => 1.7.7
msw: ^2.4.9 => 2.4.9
orval: ^7.1.1 => 7.1.1
svelte: ^4.2.19 => 4.2.19

@EddTally sure if you want to submit a PR and test it I am fine with that!


Upload data after processing to HuggingFace

The cm2.6 data gets processed, and the output is significantly smaller, but still a reasonable size. We will add this to HF as well.

https://huggingface.co/datasets/M2LInES/gfdl-cmip26-gz21-ocean-forcing

This is done -- I'll add the above link to the readme.


add solidly v3 helper

Codecov Report
Attention: 46 lines in your changes are missing coverage. Please review.

Files
Coverage Δ

contracts/SolidlyV3Helper.sol
0.00% <0.00%> (ø)

:loudspeaker: Thoughts on this report? Let us know!.


Access token not verified in the credential endpoint implementation

When looking at the credential endpoint implementation in the issuer-rest package, I couldn't find any code related to the validation of the access token generated in the access token endpoint.
This endpoint should verify the bearer authorization token passed in the header.
I might be missing where this is happening, so in that case, please point me to the file where this is handled (I'm looking for some util functions I may be able to re-use for the Credo endpoint implementation)

Yeah, totally forgot about it, it seems 🤦


Support heightmaps on Ogre 2

Terrain support is being added for Ogre 1 on #180 . The implementation for Ogre 2 is being left empty for now.
Here's a tutorial using terrain on Ogre 2 upstream:
https://github.com/OGRECave/ogre-next/tree/v2-1/Samples/2.0/Tutorials/Tutorial_Terrain

the terrain component is not enabled by default when building ogre2 so I believe our ogre-2.1 debs do not ship the terrain component. Last I tried, I ran in to a build error compiling the ogre2 terrain component on ubuntu bionic but the ogre2 dev says it should be a finished product.

We talke this via video conference and email, but just in case it wasn't clear or the information got lost:
The sample at Samples/2.0/Tutorials/Tutorial_Terrain implements the terrain system entirely (aka Terra) in a few C++ and shader files and does not depend on the Terrain Component.
The terrain component is currently abandoned.
The reason for this being is that Ogre 2.1+ targets newer rendering APIs which allow for much simpler yet more powerful terrain systems; for example Terra does not even have vertex buffers. The vertices are made procedural and the only memory it needs (excluding colouring/shading) is the heightmap.
This was impossible in older APIs, which led to much higher complexity in the Terrain Component where a lot of work went into managing vertex buffers of the current pages and LODs.

The sample at Samples/2.0/Tutorials/Tutorial_Terrain implements the terrain system entirely (aka Terra) in a few C++ and shader files and does not depend on the Terrain Component.

ah thanks for clarifying this. In that case, we'll need to copy this Terra directory and relevant shader files over to ignition?

Yep. That is correct.

Yep. That is correct.

This is great to know! So we can add heightmap support to earlier versions of Ignition as well :smile:

I'm tackling this issue and I have a few questions, based on my analysis of the ogre1 version:

ImageHeightmap is in 8 bpp! (the only implementation I could find for HeightmapData) This is rarely enough unless the size_in_km / resolution ratio is really big. Otherwise you get terrains with mountains that look like staircases. Usually 16bpp (R16_UNORM) is enough. I guess users can plug their own custom version of HeightmapData? Is that it?
Ogre1 contains minElevation because the old Terrain component had a bug if the terrain's height data had negative heights; so the terrain is offsetted as an object instead.

Ogre2's Terra needs a similar workaround because it was written assuming most heightmap sources are UNORM (it supports floats), therefore no height value in the height data is below 0
However minElevation is always set to 0!
I'm also confused because in point 1 it's clear you don't use anything that'd give negative heights since ImageHeightmap is in 8 bpp (aka R8_UNORM)

HeightmapData::FillHeightMap has _subSampling to increase the resolution. What's the rationale behind it?
OgreHeightmap split the terrain into multiple smaller terrains if the resolution was higher than 4096x4096 due to some LOD bug apparently. Terra has no such problem; however all GPUs have a hard cap on max texture size of 16384x16384 (which in VRAM would consume 1GB for heightmap + 1GB for normal data + 1GB for shadow data) is that a problem? IMHO at such large sizes, dynamic streaming (e.g. "infinite terrain") at 8192x8192 makes more sense.

Remarks
Now I understand @iche033 insistence on questions about LODs and paging. It seems Ogre1 did a lot of work to cache:

Vertex geometry
LODs
Baked textures

Terra has no such things. Everything that was baked either no longer exists (Terra has no vertex geometry, and without it, LODs are trivial as well) or is calculated in realtime.
This is like a hover car using antigravity devices and asking what happens when you get a flat tire: This is a hover car, it has no tires :smile:
Integration
The way I set it up was to copy-paste the Terra files into your own projects, and modify what you need (although for upgrade-ability sake, keep them to a minimum or submit contributions to upstream)
But I noticed the coding standards and naming conventions (i.e. including project folder structure, CamelCase, etc) are too different from Ignition. I think it may be best if we build a small library and ign-rendering links to it.
However where would that library live? as part of https://github.com/ignitionrobotics like sdformat? as part of ign-rendering/terra?
Minor challenges

Terra assumes it's rendered directly to screen by the main camera. This is a minor inconvenience where we have to update each Terra instance for each different camera (e.g. for each sensor) that will render it.
Terra is Y up. I've modified Terra to be Z up in the past. I'll try to think of ways to make this tweakable from upstream. This is annoying more than anything (because subtle bugs are hard to spot and may be hidden for a while).

ImageHeightmap is in 8 bpp!

We could update that class to also accept 16bpp.

I guess users can plug their own custom version of HeightmapData?

An alternative way to get heightmaps in is through the Dem class (https://github.com/ignitionrobotics/ign-gazebo/issues/235). I didn't add support to that while porting the Ogre 1 code for expediency, but we should eventually support it.

HeightmapData::FillHeightMap has _subSampling to increase the resolution. What's the rationale behind it?

I'm not completely sure, but it may be important for physics, which also uses that class to import heightmaps.

This is a hover car, it has no tires smile

:rocket:

naming conventions ... are too different from Ignition.

We can include it in the codebase as "vendored" code, which we usually don't run linters on (like GTest, Remotery, etc).

However where would that library live? as part of https://github.com/ignitionrobotics like sdformat? as part of ign-rendering/terra?

I think it can be part of ign-rendering. We don't need to expose it to downstream users if it will only be used internally.
I think another potential place for it could be our Ogre 2 fork?

It seems Ogre so far has been compiled without JSON support. Terra needs it.

I don't think Ogre uses ign-cmake, but maybe FindJSONCPP.cmake helps?

Terra needs heightmap data to be power of 2 (e.g. 512x512), while old Terrain needs it to be power of 2 + 1 (e.g. 513x513)

Maybe all we need is to update some documentation? Like this one: https://github.com/ignitionrobotics/ign-common/blob/465bb7780e552b278de559fafaffe3d061a99782/graphics/include/ignition/common/Dem.hh#L74-L81

I'm also confused because in point 1 it's clear you don't use anything that'd give negative heights since ImageHeightmap is in 8 bpp (aka R8_UNORM)

We mainly use ImageHeightmap to create simple terrains. For more accurate terrain simulation, we would import DEMs from online GIS sites. Sometimes we find that the DEM data contain negative heights.

HeightmapData::FillHeightMap has _subSampling to increase the resolution. What's the rationale behind it?

I'm not completely sure, but it may be important for physics, which also uses that class to import heightmaps.

We also had a heightmap editor once in gazebo-classic and increasing the resolution allowed us to make finer modification to the terrain

OgreHeightmap split the terrain into multiple smaller terrains if the resolution was higher than 4096x4096 due to some LOD bug apparently. Terra has no such problem; however all GPUs have a hard cap on max texture size of 16384x16384 (which in VRAM would consume 1GB for heightmap + 1GB for normal data + 1GB for shadow data) is that a problem? IMHO at such large sizes, dynamic streaming (e.g. "infinite terrain") at 8192x8192 makes more sense.

We've used 16K resolution before on more beefy machines with ogre 1.x. So it should be fine. For that particular project the terrain size is actually not very large, 300x300m, it just needed really high fidelity for physics simulation, and we used ogre 1.x's LOD so we can get reasonable framerate. That's one extreme case. I think dynamic streaming would be beneficial for more projects actually but we should still keep the ability for users to use 16K if they choose to.

I think it can be part of ign-rendering. We don't need to expose it to downstream users if it will only be used internally.

yeah let's put it in ign-rendering.

It seems Ogre so far has been compiled without JSON support. Terra needs it.

I don't think Ogre uses ign-cmake, but maybe FindJSONCPP.cmake helps?

ok so we'll need to add rapidjson dependency to ign-rendering, e.g. through a new cmake module in ign-cmake like FindRapidJSON.cmake. How about when packaging and releasing ogre 2.x debs, do we need to build it with rapidjson?

Terra assumes it's rendered directly to screen by the main camera. This is a minor inconvenience where we have to update each Terra instance for each different camera (e.g. for each sensor) that will render it.

ah ok, sounds like there'll be some performance hit here.

Terra is Y up. I've modified Terra to be Z up in the past. I'll try to think of ways to make this tweakable from upstream out of the box. This is annoying more than anything (because subtle bugs are hard to spot and may be hidden for a while).

:+1:

An alternative way to get heightmaps in is through the Dem class (ignitionrobotics/ign-gazebo#235). I didn't add support to that while porting the Ogre 1 code for expediency, but we should eventually support it.

Gotcha. That's all I needed to know.

However where would that library live? as part of https://github.com/ignitionrobotics like sdformat? as part of ign-rendering/terra?

I think it can be part of ign-rendering. We don't need to expose it to downstream users if it will only be used internally.
I think another potential place for it could be our Ogre 2 fork?

Ideally the changes in the fork get to upstream to reduce the need for a fork at all :)

Terra needs heightmap data to be power of 2 (e.g. 512x512), while old Terrain needs it to be power of 2 + 1 (e.g. 513x513)

Maybe all we need is to update some documentation? Like this one: https://github.com/ignitionrobotics/ign-common/blob/465bb7780e552b278de559fafaffe3d061a99782/graphics/include/ignition/common/Dem.hh#L74-L81

We'll see what we can do. In theory we could just ignore the last row and column. Or perhaps that introduces problems. Once we get there we'll evaluate. I was pointing it out.

We've used 16K resolution before on more beefy machines with ogre 1.x. So it should be fine. For that particular project the terrain size is actually not very large, 300x300m, it just needed really high fidelity for physics simulation, and we used ogre 1.x's LOD so we can get reasonable framerate. That's one extreme case. I think dynamic streaming would be beneficial for more projects actually but we should still keep the ability for users to use 16K if they choose to.

1.83 cm per pixel, impressive.
Yeah Terra can do that.

ah ok, sounds like there'll be some performance hit here.

Nah, no perf hit. I was just pointing it out; it is a minor inconvenience because all the rendering cameras (e.g. Ogre2GpuRays, Thermal, etc) need to call a function that iterates through all Terras and updates them with the current Camera.
Might also be fixed with a Workspace listener. I'm undecided on the best approach yet.

Some progress to get everyone excited.
Alignment / size is not exact to Ogre1 yet. Left is Ogre2, Right is Ogre1

'heightmap_bowl.png' is an edge case scenario for Terra (extremely artificial, non-natural terrain; looking the terrain from outside rather than the inside; and being looked from the outside without having more terrain in its surroundings), which is why the skirts and LODs are so obvious;

I'm close to finishing it and it looks they way I'd expect to given the input parameters (btw we support much more to make it more beautiful...) but I don't understand why it looks so different from reference in ogre1 (actually reference looks weird...)

What's missing:

[ ] Apply same blending mode to roughness and metalness params as diffuse
[ ] Fix edge case of skirts which looks really bad but easy to fix
[ ] Non-deterministic bug where the last N pixels of the heightmap are garbage. Looks like uninitialized variable but I don't know where it's coming from

Regarding the last one (the non-deterministic bug) some runs it looks like this:

I managed to look it like reference:

The only weird thing is that in order to do that, I had to disable a feature. It appears ogre1 is ignoring the alpha channel of the source textures; while ogre2 was using them.
I'm not sure what's the intended scenario but considering the source textures do have alpha channel, it looks it the idea was for it be used.

Progress with the non-deterministic bug: It's not mine. It's gazebo's.
Sometimes ImageHeightmap::FillHeightMap will return garbage in the last 3 pixels of the row 0.
I've printed the last 10 pixels of row 0; it should be all 0s:
0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.533333, 0.988235,
I suspect two possibilities:

ImageHeightmap::FillHeightMap was never tested against against a power of 2 output (it's always been pow of 2 + 1)
Disk access race condition: I've seen something weird: I replace city_terrain.jpg (and heightmap_bowl.png) with my pow-2 version, but after a single run, sometimes it's randomly been overwritten back to its pow of 2 + 1 (WTF?). Likewise sometimes when I restore it back to its pow2+1 version; it randomly reverts back to pow2 version (WTFx2). Is there some write access code that could cause it?

OK to recap what I'm now missing

[ ] File PR on ignition/common about the Heightmap corruption and leak
[ ] Ask whether we should honor alpha channel's content
[ ] Integrate PbsListener so other objects can receive shadows
[ ] Ask if this CMake script is ok

Ask whether we should honor alpha channel's content

chances are that we just didn't use the alpha channel with ogre 1.x. We copied shader material generation code from ogre 1.x and tweaked it to fix some issues with glsl shaders some time ago. I haven't paid much attention to the blending logic but a quick look seems to suggest that the alpha channel is used to store specular map? https://github.com/ignitionrobotics/ign-rendering/blob/ign-rendering5/ogre/src/OgreHeightmap.cc#L2275-L2291
The latest ogre 1.x doc also talks about merging the diffuse and spec textures into one texture: https://ogrecave.github.io/ogre/api/latest/tut__terrain_sky_fog.html
I think we should respect the alpha channel in ogre 2.x implementation.

Ask if this CMake script is ok

At first glance, I think it may be better to have the terra folder live inside ogre2/src directory as it's very specific to ogre 2. It's fine to not use the ignition-cmake macros to build it for now. As an example, we put an external project, Remotery, in the ign-common/profiler/src directory.
@mjcarroll, do you have any recommendation on the best way to include some ogre2-related external source files (so we can also make windows and bazel happy)?

Update 4: It corrupts because it's trying to swap a greyscale image. There's nothing to swap.

ahh.. :( looks like there needs to be a check for image format before trying to swap..

Yes. But please note it's also leaking because a copy of the freeimage handle is made which is never freed (the handle is lost)

Done in #386 :tada:


RxJava 2 support?

This is a pretty sweet little library, it's very similar to something I've been working on myself.
But RxJava 1 would makes me sad when inter-oping with the rest of my application code.
Any plans to update to RxJava 2?
The signifigant performance gains, Reactive Streams spec compliance, and Flowable vs Observable distinction alone makes it worth updating imo.
I would be willing to contribute or start a PR if there is interest.

I was hoping to wait until some larger libraries like Retrofit make the plunge. Is it just  the public api you want to change or all rxjava1 to 2? I'm worried that current adoption of rx2 is low.  Will keep open

Retrofit already supports RxJava 2 via an adapter.
My vote goes for changing everything to Rx 2 as keeping both dependencies increase the method count quite significantly. It could also be a solution that abstracts away the Rx dependency from this library altogether (e.g., using adapters).

@scottmeschke I'd be open to figuring out a way to abstract RxJava stuff into adapters so that both Rx1 & Rx2 will work, I don't believe this would be trivial but at the same time possible.  Feel free to play with it (along with anyone else that wants rx2 support before we convert internally)

I'd just make the transition to the latest version of RxJava and tell people with Lacey m legacy to use the interop library.
Rx2 is the future.

tell people with old versions to use the interop library
Is there any interop solution, which allows converting RxJava2 to RxJava1 code? As far as I know RxJava2Interop does this thing the other way around.

Both ways @pwittchen

We are making the jump to rx2 internally. Will give this a whirl soon. If anyone wants to convert feel free to start ☺

Does this go together with #22

Yeah I guess its going to have to since some operators return single by default.. Good question!

@digitalbuddha What about lambdas? With retrolamba it just compiles to bytecode like everything else and that would make the work way easier!

My only concern is the additional methods that are added. Last I checked retrolambda turned each call into 3 methods.

There is no need to add Retrolambda as an extra dependency. No obvious advantage of using it.

So I think what @PaulWoitaschek  is getting at is that the refactoring is much easier if we are dealing with Lambda's rather than having to refactor each anonymous class to be a different class.

Are you planning to drop support for RxJava v1 completely in one of the next releases?
Maybe you could create a new branch for only RxJava v2 version so others can send PRs against it?

My thought was to make 1x branch Deprecated in few releases. Then do only new development on 2.x.  If something is critical I guess it can be done in the other branch too. Like mentioned above there is an interop libraries plus it seems square is starting to convert their libs.  Thoughts?

Sounds good. So, you want to make this changes directly in develop branch, and create a new branch for RxJava v1 version, right?

Yup that sounds reasonable.

there is an experimental branch in progress.  We have another issue for tracking #167


"[esri.core.Accessor] Accessor#set Invalid property value" and "[esri.views.2d.layers.FeatureLayerView2D] TypeError: Cannot read property 'constructor' of undefined"

Another issue with the webpack demo at https://github.com/Esri/jsapi-resources/tree/master/4.x/webpack/demo - both sample layers are returning the error message:

Logger.js:25 [esri.core.Accessor] Accessor#set Invalid property value, value needs to be one of 'esri.symbols.SimpleFillSymbol', 'esri.symbols.PictureFillSymbol', 'esri.symbols.PictureMarkerSymbol', 'esri.symbols.SimpleLineSymbol', 'esri.symbols.SimpleMarkerSymbol', 'esri.symbols.TextSymbol', 'unknown', 'unknown', 'unknown', 'unknown', 'unknown', 'esri.symbols.CIMSymbol', or a plain object that can autocast (having .type = 'simple-fill', 'picture-fill', 'picture-marker', 'simple-line', 'simple-marker', 'text', 'line-3d', 'mesh-3d', 'point-3d', 'polygon-3d', 'web-style', 'cim')

In the case of this demo the layers seem to draw OK despite the error message, but in my example I'm seeing this additional error message and the feature layers aren't drawing at all.

[esri.views.2d.layers.FeatureLayerView2D] TypeError: Cannot read property 'constructor' of undefined
at Object.s [as getOwnClassMetadata] (metadata.js:25)

let layer3 = new FeatureLayer({
  url: "https://services7.arcgis.com/nMBRP09Dx4VqvXeZ/arcgis/rest/services/Indigenous_Locations/FeatureServer/0/",
  renderer: {
    type: "simple",
    symbol: {
      type: "simple-marker",
      style: "circle",
      color: "#6e6e6e",
      size: "9px",
      outline: {
        color: [ 255, 255, 255 ],
        width: 1
      }
    }
  }
})
map.add(layer3)

Any suggestions? thanks

At 4.17 including '3d': false, as a plugin feature breaks 2D feature layer symbols.
https://github.com/Esri/jsapi-resources/blob/master/4.x/webpack/demo/webpack.config.js#L78

Don't set 3d: false for now in the plugin. There are some cross dependencies for 2d and 3d, it's when it excludes all 3d here.
https://github.com/Esri/arcgis-webpack-plugin/blob/master/lib/features.js#L63
You can add your own excludes of stuff like SceneView if you want, but full 3d exclusion won't work right now. Should be fixed in next release of the plugin.

@odoe thanks for the info. However, I'm still seeing the errors despite the changes that you made to the repo. Are you also seeing them?
Any further suggestions? Thanks

No, I'm not getting that error, even with the layer3 snippet you provided above. I get a font error, but only because a font is used not available on Online, but it uses a fallback.
Did you remove the "3d": false from the webpack config?

If you're not seeing the error I'll close this for now, as I'm actually using a different approach on my current project. Thanks for the fast responses


Fix time duration calculation in test_resnet.cc

Looks line nanosec duration can not be calculated correctly for large durations. -  large than several seconds on 32 bit platforms

Change included in latest force-push.


Segmentation Fault when running in headless mode

Hi, thanks for the great works!
I followed the installation readme and I then ran this script to begin the training
scripts/teacher_baoding.sh 0
I try to visualize the environment so I keep headless=False
However, I'm getting the following error. It works normal if I set headless=True.

scripts/teacher_baoding.sh: line 15: 2849495 Segmentation fault      (core dumped) CUDA_VISIBLE_DEVICES=${GPUS} python ./isaacgymenvs/train.py headless=False task.env.objSet=ball task=AllegroArmMOAR task.env.axis=z task.env.numEnvs=8192 train.params.config.minibatch_size=16384 train.params.config.central_value_config.minibatch_size=16384 task.env.observationType=full_stack_baoding task.env.legacy_obs=True task.env.ablation_mode=no-pc experiment=baoding

I'm using Ubuntu 22.04 with Isaac Gym version 4.
Thanks in advance!

It seems likely that this issue is related to the VRAM limitations of your GPU(you can check by watch nvidia-smi). Since headless=False, the visualization GUI requires additional resources.
To address this, if you want to keep the GUI window open, you should reduce the values of task.env.numEnvs and train.params.config.minibatch_size. For example, try halving them:
task.env.numEnvs=4096 and train.params.config.minibatch_size=8192.
This adjustment should help alleviate the VRAM demand.


Remove k command and use dnx instead

https://github.com/aspnet/dnx/issues/1473

:shipit:

Most repos just need one-line-change like this. I assume I get sign-off for all of them. I am still working on some repos that use more k command than this one. Will mention you in those PRs.

Yes


Please add a new outlet device cuco.plug.cp1m

I now add the cuco.plug.cp1m device model as cuco.plug.cp1 and it works fine, but the new device cuco.plug.cp1m has the LED control function

Will add full support for that outlet in the next update👍

Support is now added as of version 0.9.17

In the latest version, cuco.plug.cp1m can be controlled, but the switch status is not obtained in time. Mihome displays on, but homekit displays off

You would need to post a homebridge debug log so I can see if there is any kind of an issue.

log.txt

You got timeouts, try to use micloud if it gets better or maybe  reduce the polling interval


fix: move bl and lru-cache types to dependencies

This PR fixes a typescript related issue raised in https://github.com/eclipse/thingweb.node-wot/issues/728 where the types for the bl and lru-cache modules could not be found during transpilation.

I think something is off, @type/x is meant to be used as devDependecies and everything should work correctly. Because in practice you distribute only the js file and the d.ts types.

Hmm, I think at least bl is an exception to the rule as OutgoingMessage extends/implements BufferList from bl. Therefore, the ts transpiler needs the type information to work. As the types are included as a devDependency here, node-wot can't access the types, though. That is the reason the ts transpiler is complaining, I think.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.1%) to 87.247%

Files with Coverage Reduction
New Missed Lines
%

lib/server.ts
1
82.48%

Totals

Change from base Build<PHONE_NUMBER>:
-0.1%

Covered Lines:
1141

Relevant Lines:
1258

💛  - Coveralls

so we sta with types for bl AND lru-cache in deps?

Hmm, yeah, there was also an error thrown for lru-cache. I just noticed that here we have a similar case, as the CoapLRUCache extends LRUCache. So I think both have to go into the regular dependencies.
We could try removing @types/node by the way, not sure if anything directly depends on a definition from the module, though.

I now removed @types/node from the regular dependencies. Transpiling in dependent projects should still work (tested out using npm install --production).

@Apollon77 Could you release a new version after merging this PR? :)

DO you want to add the version increases to the PR already? ;-)

DO you want to add the version increases to the PR already? ;-)

Sure, done :)

Is there any blocking issue left?

@danielpeintner just me having time to merge and release it together with an other fix ...


Scale in holograms dont work

Hi, first of all, sorry for my bad English, I wanted to comment that I'm trying to use the command to change the scale but it doesn't work, I already tried to restart the server and everything, I'm attaching a video for you to see
https://user-images.githubusercontent.com/76601496/232235601-f945c175-0b6d-4776-8424-254a8b3d3c5f.mp4
Plugin version: 1.0.5
Software: Paper 1.19.4 | git-Paper-510
another plugins: only FAWE and your plugin installed
edit: in the console i dont have any error

Fixed, the bug is weird, but I THINK it is not compatible with lunar client, I entered without lunar client to test something else and I saw the hologram with the scale that I put on it, and I put it back on and it looked the same small, solved then


Add Sender Configuration to Senders

In order to allow for a sender to have hard-coded values that get added into the Report object on ingestion.
Two examples are:

Having a processing_mode_code set for senders who are still building
Manually setting a CLIA or a lab name for a sender who has a schema that doesn't include it

We should add a Sender Configuration object on senders, in the same fashion we have TranslationConfiguration for outbound translations. It could potentially even inherit from TranslationConfiguration.

Moving to In Progress to start working on this.

I used the field default feature as a workaround for the absence of this feature.  It is ok, but not perfect.   The main problem is that flipping a sender from "T" Training mode in production to "P" production mode in production currently requires a code release since its in the .schema, not in the settings.
Here's an example of how I do this for a Sender called MoHealth:
---
name: mohealth-covid-19
description: Every Day we provide Mo and Mo Health
topic: covid-19
trackingElement: specimen_id
basedOn: covid-19
extends: direct/direct-covid-19

elements:
  - name: sender_id
    default: MoHealth

  - name: processing_mode_code
    default: T


Add support for EKS clusters

Objective:
Add support for aws_eks_node_group
Example output:
// TODO
Pricing details:
//TODO
Status:

[x] Added to resource_registry.go
[ ] Added resource file
[ ] Added integration tests
[x] Added unit tests if applicable - Not applicable
[x] Added to https://github.com/infracost/docs/blob/master/docs/supported_resources.md
see https://github.com/infracost/docs/pull/9

Issues:
Closes #82
Useful links:

Pricing: https://aws.amazon.com/eks/pricing/
Terraform: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group

This resource has a block like this:
scaling_config {
    desired_size = 2
    max_size     = 3
    min_size     = 1
  }

For now I should only consider desirde_size, like we did for aws_autoscaling_group yeah?

@m-yosefpor any update on this? I'd be happy to take it if you're ok with that?

Oh sorry I've forgot about it.. I'll finish it this weekend

@m-yosefpor as discussed, @sashman has kindly offered to take https://github.com/infracost/infracost/issues/82, so I'll close this.


Necessary needed python or python-is-python2 can't use with python3

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2020.05.03
[ ] I've checked that all provided URLs are alive and playable in a browser
[ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

/usr/bin/env: ‘python’: No such file or directory

Description

It necessary needed python 2 it can't be run on python3 pls bump the python dependencies.

How this is invalid.

Yup it's invaild i got this.


WebGPU bindings are broken in 2.21.0

Summary
As far as I can tell, the WebGPU bindings are not correctly marshalling RenderPassColorAttachment.LoadOp (and possibly others, like StoreOp).
On 2.20.0, it works as expected.
On 2.21.0, the WGPU implementation panics with:
thread '<unnamed>' panicked at src/lib.rs:1184:38:
invalid load op for render pass color attachment: 0
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
   4: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
   5: _wgpuCommandEncoderBeginRenderPass
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5

Steps to reproduce
Here is a minimal reproduction, which is supposed to clear the backbuffer with a magenta colour: https://gist.github.com/smoogipoo/0e248fe961cc3f42930c6091eb1bd18e

Platform: macOS, desktop
Framework Version: .NET Core 8
API: WebGPU / WGPU

This is due to a mismatch between the headers and the implementation in WGPU. Should be fixed by new binaries + a regen.

Is there any schedule on releases? We're trying to implement WGPU in our project to test the viability of Silk.NET going forward and this is a blocker because the 2.20 impl appears to be causing shader/uniform validation errors (not Silk.NET's fault) which look to not be happening in 2.21.

Is there any schedule on releases? We're trying to implement WGPU in our project to test the viability of Silk.NET going forward and this is a blocker because the 2.20 impl appears to be causing shader/uniform validation errors (not Silk.NET's fault) which look to not be happening in 2.21.

We try to release monthly, and its almost been a month since the last release, so if everything goes to plan, we'll try to get a release out this week

Is there any schedule on releases?

So the answer to this question is very complicated and I don't think I've put this on GitHub before (only Discord) so I'll do it now. Silk.NET originally aimed to have an update every Patch Tuesday (used to be every 1st Friday before that, but this resulted in us releasing too early to get the latest of some dependencies e.g. the Vulkan headers). However, the first month we did this we discovered some generator issues that essentially rendered everything but the Khronos bindings unmaintainable except on one blessed VM image where it works correctly which is in Beyley's possession.
Essentially, libclang made some breaking changes which we are not prepared to adapt to which resulted in us abstaining from migrating to the latest ClangSharp. All was fine for a while. Then, the MSVC headers updated to prevent usage on anything but the latest clang. Not ideal. This was worsened by the fact that people's dev machines seemed to be updating at random intervals, resulting in basically all maintainers and our selfhosted CI runner being unable to regenerate anything but the Khronos bindings (which generate from XML files instead of C header files).
Beyley's VM isn't online long enough to update, and is basically what we're reliant upon atm. It's possible that we could figure out a way to get this uploaded onto Azure so it's not just Beyley's dev machine that has this VM.
As a result, we're trying to focus all maintainer effort towards 3.0. However, maintainer effort is basically non-existent. We don't have enough people for the amount of time each person can commit.
I've been working hard on our new generator, which is based on ClangSharp P/Invoke Generator (which is subject to the same breaking changes but has an easier time adapting to them due to its architecture), but this is just one part of a very large library.
So the TLDR is: supposed to be monthly, but the entire project is resting on a single person's shoulders and the effort to rectify that is also resting on a single person's shoulders. We're maintaining as much as we need to such that the library remains in a usable and recommendable state, but anything more is building on sand at this point.
This has been discussed at length in our coffee & codes and API reviews on the .NET Foundation YouTube channel.
Aside: I have other commitments starting 1st June that could possibly tie me up for 6 months, so that would reduce all maintenance effort down to just Beyley. My aim now that 3.0 OpenGL codegen is basically finished is to prototype my ideas for everything else such that the community can hopefully take over.
We're stable for now and still updating, and will aim to have a release out as soon as possible. I have a personal commitment to not let this library rot, so that's not at risk of happening, but multi-month periods of no updates wouldn't be entirely unexpected given that we tend to release once enough changes have piled up or we have users blocked on something. This would constitute the latter, and thankfully we're both in a position to make this specific blocker right at this current moment :)

For anyone who (like me) who is seeing very broken rendering after downgrading Silk.NET.WebGPU.Native.WGPU to 2.20.0, make sure to also downgrade Silk.NET.WebGPU and Silk.NET.WebGPU.Extensions.WGPU to the same version as well.


MD5 identified as GUID - how to avoid ?

When I feed in an MD5 hash e.g. "md5": "57920d1feee527313658340bb1ab101a"
It is parsed as "md5_g": "57920d1f-eee5-2731-3658-340bb1ab101a".
How to avoid the reformatting ?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1295e1b3-5aed-3aee-4331-8b81e7a7e544
Version Independent ID: 3c00a5ee-7093-1e8b-9004-8fed3f82960e
Content: Azure Monitor HTTP Data Collector API - Azure Monitor
Content Source: articles/azure-monitor/platform/data-collector-api.md
Service: azure-monitor
Sub-service: logs
GitHub Login: @bwren
Microsoft Alias: bwren

Though not for MD5's, this same thing is impacting me.  I'm loading data into a log analytics workspace so that I can join with data in application insights (very similar use case to examples provided in the documentation).  The value I'm joining on is a string in application insights, without dashes, but log analytics is converting it to a guid (sometimes), so it's with dashes on ingestion so I can not easily perform the join.

assigning to @bwren for input.

I've run into the same concern.  I want to be able to join with Application Insights by operation id:
app("my-app").requests
| join kind=inner (
workspace("my-app").TestTable_CL
) on $left.operation_Id == $right.operationId_s

If the operationId value is converted to guid formatting, then $left.operation_Id == $right.operationId_g never matches.
I tried forcing the creation of operationId_s by first submitting a record with a string value that can't be cast to a GUID, but then subsequent submitals with actual operationId values still create operationId_g.

This is now noted in the article. #please-close


Improve show syntax tree

This implements some of the features discussed in #820.
You can now select a range of syntax in a file and then use "Show Syntax Tree" to show its syntax. In addition you can select a range of syntax that is inside a string (typically test cases) and show its syntax as well.
Previous behavior is still available, simply use "Show Syntax Tree" without a selection, and you get the live updating syntax tree. Additionally now the live updating tree will update when the active file is changed. Previously you had to type something in the new file to get the syntax tree to update.

bors r+

In addition you can select a range of syntax that is inside a string (typically test cases) and show its syntax as well

:heart_eyes:


Remove kprobe legacy logic once upstream accepts it

Discussing libbpf: add legacy kprobe attach support at:
https://github.com/libbpf/libbpf/issues/317
libbpfgo "cgo: attach_kprobe_legacy" and "doAttachKprobeLegacy" logic should be removed once libbpf accepts that patch.

Latest proposal at LORE thread.

Latest proposal at:
<EMAIL_ADDRESS>already to be applied on top of the:
[PATCH v3 bpf-next 00/14] BPF perf link and user-provided bpf_cookie
patch set from Andrii. It will need to wait until that patch set is merged. It correctly replaces libbpfgo legacy kprobe implementation embedded at libbpfgo.go source code (for current libbpf upstream branch).

Commit got merged in for-next (bpf-next):
commit 467b3225553a (HEAD -> rafaeldtinoco, origin/master, origin/for-next, origin/HEAD)
Author: Rafael David Tinoco<EMAIL_ADDRESS>Date:   Sun Sep 12 03:48:44 2021

    libbpf: Introduce legacy kprobe events support

    Allow kprobe tracepoint events creation through legacy interface, as the
    kprobe dynamic PMUs support, used by default, was only created in v4.17.

    Store legacy kprobe name in struct bpf_perf_link, instead of creating
    a new "subclass" off of bpf_perf_link. This is ok as it's just two new
    fields, which are also going to be reused for legacy uprobe support in
    follow up patches.

    Signed-off-by: Rafael David Tinoco<EMAIL_ADDRESS>    Signed-off-by: Andrii Nakryiko<EMAIL_ADDRESS>    Link<EMAIL_ADDRESS>
It will get into bpf-next and, eventually, into libbpf (github). We will be able to get rid of legacy kprobe functions.

Alright, the legacy kprobe logic just got merged into libbpf:
https://github.com/libbpf/libbpf/commit/749b3942a09300a570198bc27814405ab92607ae
and into bpf-next tree. Andrii has already refactored it to support legacy uprobes as well as some fixes.
We're ready for the legacy kprobe specific functions removal in libbpfgo.

I think we should wait with that until libbpf 0.6.0 (which should include the patch) will be out

@rafaeldtinoco I think we can now remove those functions, can't we?


Upgrade docker images for distributed testing to unbutu:16.04

https://ci.tensorflow.org/job/tensorflow-pull-requests-mac/3805/console shows an interestng new error I haven't seen before -- do you think it's related?

Can't possibly be related since we don't even have Docker installed on the macs. Looking at the errors, it seems that someone changed the numpy version on our macs.

@vrv. I don't think it is related. The files in tools/dist_test is not depended on by the failing tests.
Looking closer at the log, I noticed that most of failures have the following error:
TypeError: norm() got an unexpected keyword argument 'keepdims'

as well as:
TypeError: linspace() got an unexpected keyword argument 'dtype'

These seem to be some sort of numpy version inconsistency in the functions np.linalg.norm and np.linspace on that particular Mac machine (mac0)

@yifeif another instance of inconsistency ...

@yifeif  can you check the version of numpy and scipy on mac0?
On mac1, they are 1.11.0 and 0.15.1, respectively, consistent with our slave install script.

numpy.version.version
'1.11.0'
scipy.version.version
'0.15.1'

but looks like builds on mac0 is running with different Python library path compared to other macs(/Library/Python/2.7/site-packages vs /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages)??
I'll take it offline and investigate further.


Drop support for 0.4

Codecov Report

Merging #49 into master will decrease coverage by 0.17%.
The diff coverage is 96.07%.


Notebooks to Implement

Introduction

[x] An Introduction to Julia
[x] Getting started with JuMP
[x] Variables, Constraints and Objective
[x] Solutions
[x] Solvers

Using JuMP

[x] Reading from Data files
[x] Problem Modification
[x] Nonlinear Modeling

Optimization Concepts

[x] Integer Programming
[x] Conic Programming
[x] Benders Decomposition

Modeling Examples (Currently a list of topics to pick examples from)

[x] Sudoku
[x] Power Systems
[x] Network Flows
[x] Statistics
[ ] Approximation and Fitting
[x] Geometric Problems
[ ] Control theory
[x] Graph Theory
[ ] Economics/Econometrics
[x] Finance
[x] Rocket Control
[ ] Travelling Salesman Problem

Note that this is a tentative list and some notebooks can be combined or split into two parts. Topics such as non-linear modelling, callbacks and column generation are ongoing dev efforts. Things are subject to change there and notebooks for them will only be created after they are finalized.

The following notebooks have been pushed back for now

[ ] Optimizing Code
[ ] Building Large Scale Models
[ ] Using Different Julia Data Types
[ ] Duality

Regarding TSP I wrote: https://opensourc.es/blog/mip-tsp a while back.
It's without callbacks just a loop of solving/disallowing cycles. If this is of interest I can add a clean version of this here.

@Wikunia yes we might take it as a basis, I think it would be a good use case to introduce callbacks, pinging @mtanneau

@Wikunia It would be great if you could add your tutorial here. I think it would be good to have both versions, with and without callbacks.

It would be really helpful to add a tutorial on plotting or graphing within Julia.  It could be at the level of your tutorial on "reading data."

Hi!
I wrote a Spacecraft Trajectory Optimizer with Avoidance Constraints program (in JuMP) using Mixed-Integer Linear programming, with at the end a simple plot of the solution.
Is there someone to contact to see if a tutorial about this topic might be useful?

Here is the right place, the contribution is welcome!

Closing because I'm going to archive this repository.


Zwift crashing at startup with "GameMode ERROR: Could not connect to bus:"

Checklist

[X] I have given details of my install including Distribution, Wayland/ XOrg, Parameters Used, echo $XAUTHORITY, etc.
[X] I have provided logs showing any errors, if available (use DEBUG=1 zwift)
[X] I have filled out the issue template to the best of my ability.
[X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
[X] This issue is not a duplicate issue of previous issue.

Describe the issue
When running a container (latest tag), Zwift will crash after briefly displaying a window with its logo. The process fails with the following error:
GameMode ERROR: Could not connect to bus: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
dbus[219]: arguments to dbus_connection_unref() were incorrect, assertion "connection != NULL" failed in file ../../../dbus/dbus-connection.c line 2832.
This is normally a bug in some application using the D-Bus library.

Distribution Details
Runtime

distro: I'm running on NixOS 24.05.
I'm using Wayland.
variables CONTAINER_TOOL=docker WINE_EXPERIMENTAL_WAYLAND=1.
echo $XAUTHORITY returns /run/user/1000/.mutter-Xwaylandauth.4IZNS2.

Debug log
The following output has been generated by running zwift with DEBUG=1:
+ [[ -f /home/gmodena/.config/zwift/config ]]
+ [[ -f /home/gmodena/.config/zwift/gmodena-config ]]
+ [[ ! -z '' ]]
+ WINDOW_MANAGER=Other
+ IMAGE=docker.io/netbrain/zwift
+ VERSION=latest
+ NETWORKING=bridge
++ id -u
+ ZWIFT_UID=1000
++ id -g
+ ZWIFT_GID=100
+ '[' '!' docker ']'
+ '[' docker == podman ']'
+ LOCAL_UID=1000
+ CONTAINER_UID=1000
+ CONTAINER_GID=100
+ case "$XDG_SESSION_TYPE" in
+ WINDOW_MANAGER=Wayland
+ '[' Wayland = Wayland ']'
+ '[' -z 1 ']'
+ WINDOW_MANAGER=Wayland
++ id -u
+ '[' 1000 -ne 1000 ']'
+ [[ ! -n '' ]]
++ curl -s https://raw.githubusercontent.com/netbrain/zwift/master/zwift.sh
++ sha256sum
++ awk '{print $1}'
+ REMOTE_SUM=732988328e5b86a1174a828a9915f225b024b235d625867a8d2fc092dc09b76f
++ sha256sum .local/bin/zwift
++ awk '{print $1}'
+ THIS_SUM=732988328e5b86a1174a828a9915f225b024b235d625867a8d2fc092dc09b76f
+ '[' 732988328e5b86a1174a828a9915f225b024b235d625867a8d2fc092dc09b76f = 732988328e5b86a1174a828a9915f225b024b235d625867a8d2fc092dc09b76f ']'
+ echo 'You are running latest zwift.sh 👏'
You are running latest zwift.sh 👏
+ [[ ! -n '' ]]
+ docker pull docker.io/netbrain/zwift:latest
latest: Pulling from netbrain/zwift
Digest: sha256:b132af52c6e05c43007c6233dd5145e853e9f4e01c7bbca6f990f79339bf918a
Status: Image is up to date for netbrain/zwift:latest
docker.io/netbrain/zwift:latest
+ GENERAL_FLAGS=(-d --rm --privileged --network $NETWORKING --name zwift-$USER --security-opt label=disable --hostname $HOSTNAME -e DISPLAY=$DISPLAY -e ZWIFT_UID=$CONTAINER_UID -e ZWIFT_GID=$CONTAINER_GID -e PULSE_SERVER=/run/user/$CONTAINER_UID/pulse/native -v zwift-$USER:/home/user/.wine/drive_c/users/user/Documents/Zwift -v /run/user/$LOCAL_UID/pulse:/run/user/$CONTAINER_UID/pulse)
+ [[ -f /proc/driver/nvidia/version ]]
+ VGA_DEVICE_FLAG=--device=/dev/dri:/dev/dri
+ [[ -n unix:path=/run/user/1000/bus,guid=5889ad23b80c43accb1a037066c65c37 ]]
+ [[ unix:path=/run/user/1000/bus,guid=5889ad23b80c43accb1a037066c65c37 =~ ^unix:path=([^,]+) ]]
+ DBUS_UNIX_SOCKET=/run/user/1000/bus
+ [[ -n /run/user/1000/bus ]]
+ DBUS_CONFIG_FLAGS=(-e DBUS_SESSION_BUS_ADDRESS=$(echo $DBUS_SESSION_BUS_ADDRESS | sed 's/'$LOCAL_UID'/'$CONTAINER_UID'/') -v $DBUS_UNIX_SOCKET:$(echo $DBUS_UNIX_SOCKET | sed 's/'$LOCAL_UID'/'$CONTAINER_UID'/'))
++ echo unix:path=/run/user/1000/bus,guid=5889ad23b80c43accb1a037066c65c37
++ sed s/1000/1000/
++ echo /run/user/1000/bus
++ sed s/1000/1000/
+ '[' Wayland == Wayland ']'
+ WM_FLAGS=(-e WINE_EXPERIMENTAL_WAYLAND=1 -e XDG_RUNTIME_DIR=/run/user/$CONTAINER_UID -e $WAYLAND_DISPLAY=$WAYLAND_DISPLAY -v $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY:$(echo $XDG_RUNTIME_DIR | sed 's/'$LOCAL_UID'/'$CONTAINER_UID'/')/$WAYLAND_DISPLAY)
++ echo /run/user/1000
++ sed s/1000/1000/
+ '[' Wayland == XWayland ']'
+ '[' Wayland == XOrg ']'
+ '[' Wayland == XOrg ']'
+ '[' docker == podman ']'
++ docker run -d --rm --privileged --network bridge --name zwift-gmodena --security-opt label=disable --hostname framework-nixos-1 -e DISPLAY=:0 -e ZWIFT_UID=1000 -e ZWIFT_GID=100 -e PULSE_SERVER=/run/user/1000/pulse/native -v zwift-gmodena:/home/user/.wine/drive_c/users/user/Documents/Zwift -v /run/user/1000/pulse:/run/user/1000/pulse --device=/dev/dri:/dev/dri -e DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus,guid=5889ad23b80c43accb1a037066c65c37 -v /run/user/1000/bus:/run/user/1000/bus -e WINE_EXPERIMENTAL_WAYLAND=1 -e XDG_RUNTIME_DIR=/run/user/1000 -e wayland-0=wayland-0 -v /run/user/1000/wayland-0:/run/user/1000/wayland-0 docker.io/netbrain/zwift:latest
+ CONTAINER=a0851f7b937ecfae8b44778f44cfd10c843145ac941929cccbf5b893c3d1079b
+ '[' 0 -ne 0 ']'
++ command -v xhost
+ '[' -x /nix/store/h6i9ar169n5p4795dsp6r04kng73y7ar-xhost-1.0.9/bin/xhost ']'
++ id -u
+ '[' 1000 -ne 1000 ']'

Reproduction steps

The latest version of the script is installed under .local/bin/zwift, with a config similar to the one described in #38
CONTAINER_TOOL=docker WINE_EXPERIMENTAL_WAYLAND=1 .local/bin/zwift
After a while, the Zwift logo appears and the container crashes.

May be caused by the screensaver inhibitor.
[[ -n "${DBUS_SESSION_BUS_ADDRESS}" ]] && while true; do sleep 30; xdg-screensaver reset; done &
Could you try to run the container interactively and change the run_script.sh file manually and run it to see if it changes anything?
Basically if you run DEBUG=1 zwift.sh you should see a line like the following in the log:
docker run -d --rm --privileged --network bridge --name zwift-netbrain --security-opt label=disable --hostname netwrk -e DISPLAY=:0 -e ZWIFT_UID=1000 -e ZWIFT_GID=1000 -e PULSE_SERVER=/run/user/1000/pulse/native -v zwift-netbrain:/home/user/.wine/drive_c/users/user/Documents/Zwift -v /run/user/1000/pulse:/run/user/1000/pulse --env-file /home/netbrain/.config/zwift/config -e DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus -v /run/user/1000/bus:/run/user/1000/bus -e WINE_EXPERIMENTAL_WAYLAND=1 -e XDG_RUNTIME_DIR=/run/user/1000 -e wayland-1=wayland-1 -v /run/user/1000/wayland-1:/run/user/1000/wayland-1 docker.io/netbrain/zwift:latest

And then simply change the -d flag to -it and add --entrypoint bash
You will also have to add vim, or your favorite editor in the container, so execute apt-get update && apt-get -y install vim
and finally edit the /usr/bin/run_zwift.sh file and run the entrypoint.sh script from within the container.
Hope this helps.

You could also just try removing -e DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus,guid=5889ad23b80c43accb1a037066c65c37

I can only suspect that gamemode has some dbus integration that isn't working on your system, try removing the DBUS_SESSION_BUS_ADDRESS environment variables as mentioned.

hey @netbrain . Thanks for the quick reply.
I tried removing DBUS_SESSION_BUS_ADDRESS, but this resulted in the same GameMode error.

Maybe try to remove gamemode altogether?

@netbrain I did try to remove gamemode and re-build the image with the bin/build-image.sh script. The image built, but it reported this error/warning:
0050:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0050:err:winediag:nodrv_CreateWindow L"The explorer process failed to start."
0050:err:systray:initialize_systray Could not create tray window
0024:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0024:err:winediag:nodrv_CreateWindow L"Make sure that your X server is running and that $DISPLAY is set correctly."
------------------------------------------------------
warning: Note: command wine NetFx64.exe /q /c:install.exe /q returned status 43. Aborting.
------------------------------------------------------

When I run the container ($ CONTAINER_TOOL=docker WINE_EXPERIMENTAL_WAYLAND=1  DONT_PULL=1 .local/bin/zwift  ) it now fails with that error too. Looks like wine is not starting properly?
I need to check what's up inside the container. I'll f/up here if I have more useful info.

Just change the contents of the container interactively, no need to build. See my steps here
https://github.com/netbrain/zwift/issues/154#issuecomment-2303861555

Just change the contents of the container interactively, no need to build. See my steps here

Ah! Actually, if I run /usr/bin/entrypoint interactively, zwift successfully starts (without dying) regardless of the gamemode dbus failure.
I tried with and without commenting out the relevant line in run_zwift.sh, and zwift starts either way.
I wonder if the demon run modes just because entrypoint returns an error. This is after the  gamemode crash:
# echo $?
134

However, it looks like dbus is not working in general though, not just for gamemode. From inside the container (and /run/user mounted):
$ dbus-monitor --session
Failed to open connection to session bus: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Might be something on my end.

Might be something on my end.

FWIW: I tried starting a dbus session inside the container, before running entrypoint, and with that gamemode does not crash. So it looks like the issue is with shared socket between the container and my host.
@netbrain many thanks for the help troubleshooting this. Do feel free to close the issue if you think it's not relevant (apologies for the noise, in case).

@perrin4869 any idea what could be causing this issue? is it a one-off or is this an actual issue other can experience?

sounds like dbus is not setup correctly?
in my system I run
if [[ -z "$DBUS_SESSION_BUS_ADDRESS" ]]; then
  export $(dbus-launch --exit-with-x11)
fi

inside .xprofile (which I run from .xinitrc via . ~/.xprofile) to start the session bus, I think @gmodena would need to figure out how dbus is being started in his system and figure out why he can't connect to the session bus. This could affect other functionality in his system. Or maybe dbus daemon crashed? You could check if dbus is running?

hey @perrin4869 .

Or maybe dbus daemon crashed? You could check if dbus is running?

On the host, I start dbus with systemd (both system and user sessions) and it seems to be working ok. I can access (read/write) the socket $DBUS_SESSION_BUS_ADDRESS from my user's Gnome session. So do all the services running in the wayland/Gnome session.
However, I can't connect from the zwift docker container using that socket. uid/gid and permissions seem match the host.
There's no dbus (host) daemon crash before/while/after the docker container is running.

Out of curiosity, could you try from podman? I personally don't use docker and I never tested it.
One possible solution is to add a check for dbus connectivity, and only call gamemode if dbus connectivity has been confirmed.
It also sounds like a bug in gamemode, they should probably handle dbus failures without crashing?

@perrin4869 I did a couple of tests with podman
The app works if I run the container with:
CONTAINER_TOOL=podman  DEBUG=1 .local/bin/zwift

Hurray!
I tested dbus manually (dbus-monitor --session & c) and everything works as expected.
However, wine fails to start  if I enable WINE_EXPERIMENTAL_WAYLAND:
CONTAINER_TOOL=podman WINE_EXPERIMENTAL_WAYLAND=1 DEBUG=1 .local/bin/zwift

The failure mode this time is different:
018c:err:combase:RoGetActivationFactory Failed to find library for L"Windows.Foundation.Diagnostics.AsyncCausalityTracer"
01fc:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
01fc:err:winediag:nodrv_CreateWindow L"The explorer process failed to start."
018c:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
018c:err:winediag:nodrv_CreateWindow L"The explorer process failed to start."
01dc:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
01dc:err:winediag:nodrv_CreateWindow L"The explorer process failed to start."
+ pgrep -f ZwiftApp.exe
+ echo 'Killing uneccesary applications'
Killing uneccesary applications
+ pkill ZwiftLauncher

One possible solution is to add a check for dbus connectivity, and only call gamemode if dbus connectivity has been confirmed.

Sounds good. If connectivity is available, the (host) daemon should ack a ping:
$ dbus-send --session --dest=org.freedesktop.DBus --type=method_call --print-reply /org/freedesktop/DBus org.freedesktop.DBus.Peer.Ping

Is GameMode the only component affected by lack of connectivity with the host?
It's weird that this issue only manifests with docker though. I still wonder if there's something messed up with my host.

It also sounds like a bug in gamemode, they should probably handle dbus failures without crashing?

I'm not familiar with gamemode, but yeah. It'd be nice if the lack of dbus access would be handled gracefully.

I'm not familiar with gamemode, but yeah. It'd be nice if the lack of dbus access would be handled gracefully.

I second this. 👍😅

gamemode is the only program accessing dbus, so adding the check would solve the issue. Maybe this can be solved upstream in the meantime though, might be worth opening an issue in the gamemode repo to request that it doesn't crash 😅
but yeah, if podman works and docker doesn't, it's most likely some permissions issue... podman has this userns flag which solves a lot of these problems. I think it's worth considering adding a note in the README recommending podman over docker, I know from personal experience the first time I tried running this repo I also tried to start with docker but ran into issues, then a few days later was surprised to find everything was smooth with podman
I never ran wayland yet, don't know about the last podman error ^^;;

oh, just looking into gamemode I found this commit: https://github.com/FeralInteractive/gamemode/commit/3fa41891cf2507928cf37878cd89cde17db27b20
@gmodena are you running the latest 1.8.2 gamemode?


Relations - Review

@phugon - here's the list of relations with the number of records under each as of 03.12. Let me know which ones you'd like to delete/combine

Relation
Number of records

PersonAuthorOfWork
879

PersonStudentOfPerson
747

WorkHasAsAnInstanceInstance
675

PersonActiveAtPlace
568

WorkCommentaryOnWork
442

PersonDirectPredecessorInLineageOfPerson
158

PlaceIsLocatedWithinPlace
132

WorkComposedAtPlace
115

PersonHasOtherTypeOfPersonalRelationToPerson
92

PersonRefersWithNameToTheViewsOfPerson
81

PersonRefersWithoutNameToTheViewsOfPerson
77

PersonOrdinatorOfPerson
71

InstanceIsCopiedFromInstance
65

PersonScribeOfInstance
55

PersonOwnerOfInstance
37

InstanceCopiedWrittenDownAtPlace
32

PersonParentOfPerson
29

PersonStudiedWork
27

PersonTeachesWork
24

PersonPrompterOfWork
19

PersonSiblingOfPerson
15

PersonRequestorOfPerson
12

WorkRecordsTheTeachingOfPerson
8

PersonBiographedInWork
7

PersonDiscipleOfPerson
7

PersonAddresseeOfWork
5

PersonBiographerOfPerson
5

PersonSponsorOfInstance
4

PersonFellowMonkOfPerson
2

PersonAuntMaternalPaternalOfPerson
1

PersonFellowStudentOfPerson
1

PersonPatronOfPerson
1

PersonPromoterOfWork
1

WorkQuotesWithNameTheViewsOfPerson
1

PersonSpiritualFriendOfPerson
1

PersonCitesWork
0

WorkContainsCitationsOfWork
0

PersonEditorOfInstance
0

PersonIllustratorOfInstance
0

PersonLenderOfInstance
0

WorkNamesPerson
0

WorkNamesWork
0

WorkQuotesWithoutNameTheViewsOfPerson
0

Also to be deleted: the relations:
WorkQuotesWithNameTheViewsOfPerson
WorkQuotesWithoutNameTheViewsOfPerson
which are now (0)

delete relation:
PersonDiscipleOfPerson
(to be continued)

delete relation
Person annotator of Instance


[Genetics] Refactor both Page components to TS

@carcruz is there a way to remove one of these? They end up being identical in styling and HTML.
There is some repetition between components and ot-ui-components (e.g. link also). What is the purpose of ot-ui-components?
Description
Please include a summary of the change and which issue is fixed. List any dependencies that are required for this change.
Issue: #2871
Deploy preview: (link)
Type of change
TS Refactor
How Has This Been Tested?
Manual, still compiles and runs
Checklist:

[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] Any dependent changes have been merged and published in downstream modules

Hey, Thanks for bringing this forward as it helps us maintain a more organized code.
components is folder inside each application that are specific to them. ot-ui-components is a shared package.
We are currently in process of moving similar components from both application like you mentioned to ot-ui-apps\packages\ui\src. This folder will serve as a repository for files that are commonly accessed and shared by both applications.
For better understanding, I would take a quick look at ot-ui-apps\packages\ui\src\LongText.tsx or ot-ui-apps\packages\ui\src\Link.tsx  and then exporting them from ot-ui-apps\packages\ui\src\index.tsx.
If you have any additional suggestions or concerns, please feel free to share them.

Thanks for the explanation Chintan! That sounds great.
Given that, what changes do I need to make to my PR for now?

Below are general steps for moving a comp into ui package.

Create a new component named Page.tsx in the path ot-ui-apps\packages\ui\src .
Check what is different between both component and try to create a common component can be used in both.
If it has some different data then pass it as props, if different style then and boolean props (this case will fit) or it has some api calls then create a context to resolve it.
Import this component and check on both apps if it is working as expected.
Check all the places it has been used(not in this case).
Delete the old component.

Please let me know if i can provide further explanation .

Hey,
While working on this issue-2903, we required to make changes in page.tsx. But unfortunately due to conflict, we were not able to merge this before start working.
But on the bright side we worked on it as planned and moved Page.tsx to UI package and importing it in both applications.
Here is the link you can check for the new file, we have added a new folder Pages in ui.
https://github.com/opentargets/ot-ui-apps/pull/156/files .
I think if you could update all the PR's with latest main, we will be good to review it after upcoming release.
Please let me know if i can help with anything.


profile::network and profile::hostname conflict on RHEL

on an RHEL system, profile::network and profile::hostname are both trying to declare file /etc/sysconfig/network.
Also note that the RHEL documentation says that the HOSTNAME entry in /etc/sysconfig/network is supposed to be the fqdn; hostname.pp does that correctly, but network.pp sets it to just the first component of fqdn.
from network.pp:
 if $::osfamily == 'RedHat' and $network_template != '' {
   file { '/etc/sysconfig/network':
       ensure  => 'present',
       content => template($network_template),
     }

from hostname.pp:
     if $update_network_entry {
          case $::osfamily {
            'RedHat': {
              file { '/etc/sysconfig/network':
                ensure  => present,
                content => "NETWORKING=yes\nNETWORKING_IPV6=no\nHOSTNAME=${calc_fqdn}\n",
                notify  => Exec['apply_hostname'],
              }
            }
            default: {}
          }
        }

thansk for the info, the quick workaround is to set somewhere on your hieradata:
network::network_template: ''

I'm thinking that longer term, networking ought to set up the file, and hostname ought to use file_line to set the name.


Allow runtime SystemData

Fixes #79
Closes #84
@tversteeg Sorry, I want to get this finished so I decided to do it myself. Thanks for your help though!

bors delegate=Rhuagh


Githug doesnt work

I have installed githug on my computer following the instructions. After installation, when I tried running the 'githug' command, I encountered an error.

It seems that exists has been removed from ruby version 3.2.0 onwards. We may have to install ruby version below 3.2.0 for the time being.  Adding the [stack overflow link](https://stackoverflow.com/a/75353113 link)

Then what is the solution for the above error ??
Should i install ruby version below 3.2.0 in my system @VarunSingh2 ,
Hey @agderes Did you found any solution for the above issue ??

Then what is the solution for the above error ??
Should i install ruby version below 3.2.0 in my system @VarunSingh2 , Hey @agderes Did you found any solution for the above issue ??

Yes, i did install ruby version below 3.2.0 but dont remember which one.


jax initialization issues

Split from #487: Pymbar 4.0 uses jax.numpy and jax is not well configured for newer macs(m1) and I tried using it on an intel i7 cpu but it is not getting initialized.

@karanv99 can you leave more information here?

Yup sure,
On MAC M1
For this diagnosis I am running Pymbar4.0.1 in a M1 mac air 2020(macOS Monterey).
I will attach a screenshot of successful installation of pymbar 4.0 for your diagnosis.

This is the error that I get when trying to import pymbar 4.0

I tried reading about it and went through your source code trying to figure out what is wrong? The most I could come up with by comparing pymbar3.0 and pymbar4.0 source code was that you in 4.0 onwards you started using jax.numpy instead of numpy arrays(probably because of efficiency or smoothnes). And I found jax has issues on apples silicon chips because of the different architecture and is not well configured.
On Inter i7 PC
Chip info

Prompt of successful installation of pymbar4.0

This is where initializing mbar using pymbar 4.0 gets stuck on the i7 chip.

I tried and waited for more than an hour, hopping it would get initialized but it was unsuccessful. I compared mbar_solvers.py between 3.0 and 4.0, in pymbar3.0 adaptive is the default solver while it is hybr in 4.0. Even though the prompt does claim to try out next method, the code gets stuck as mentioned in the image 5.
Disclaimer
I am new to this whole thing and I am trying to figure things out myself, so I might be wrong on many levels which I have mentioned above. If i am doing something silly, I really do apologise.
Thank you!! :)

So, there is no installation issue on Intel, which  is good.  "No GPU/TPU found, falling back to CPU" is exactly what you want to happen.
For pymbar 4.0 on intel, try using the solver 'robust', and setting the output to 'verbose' (see some of the examples), see what is happening, and let me know what is going on.  If there is poor overlap in the collected data, then convergence can take some playing around.
For the problems installing in M1, conda installation is the suggested route (on all applications).  It could be that the pip install is getting confused.  Can you try to install via conda instead?  Install anaconda or miniconda, set the channels as described in the documentation, and conda install pymbar there.

I don't have an M1 to test against directly, but from what I am reading of other people having the "partially initialized" is their jax and their jaxlib installs are cached from before they had better M1 support. Can you try uninstalling, clearing the cached versions of jax and jaxlib, then reinstall and see what happens?
The other main issue I am seeing is Homebrew jax and jaxlib are a bit more unreliable on M1 than pip and conda.

Hello, everyone. I have an update:
I am trying to figure things out how pymbar 4.0 works(like its syntax etc etc) on the intel PC. I will get back once I figure out with it.
As for the M1, as @mrshirts suggested I think it was the issue with the pip install. I installed pymbar4.0 using conda and it installed the required jax libraries and pymbar is getting imported. I am still figuring out how generate_fes() works on pymbar. Once I do that I will get back.
Thank you all for your help!!
Note!!
Could you please check out the output repository for parallel tempering? I think it has not been updated with the new parallel tempering code. It has the same output from the previous repo.


Can't get historical features as Dataframe

Hi there,
I am not sure if I am missing something.
But I've downgraded my feast version to 0.16.0 and used feast-postgres 0.2.4
When I try to get the historical features I get this:
 entity_df = pd.DataFrame.from_dict(
        {
            "Id": [893, 414, 523],
            "event_timestamp": datetime(2022, 2, 10, 15, 28, 28),
        }
    )

job = store.get_historical_features(
        entity_df=entity_df,
        features=["houses_features:YrSold", "houses_features:SaleCondition", "houses_features:SalePrice"],
    )

training_df = job.to_df()

Traceback (most recent call last):
  File "examples/houses_read_features.py", line 85, in <module>
    hist_features = get_historical_features(store, entity_df)
  File "examples/houses_read_features.py", line 30, in get_historical_features
    training_df = job.to_df()
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast/infra/offline_stores/offline_store.py", line 43, in to_df
    features_df = self._to_df_internal()
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast_postgres/offline_stores/postgres.py", line 204, in _to_df_internal
    return self._to_arrow_internal().to_pandas()
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast_postgres/offline_stores/postgres.py", line 211, in _to_arrow_internal
    with self._query_generator() as query:
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast_postgres/offline_stores/postgres.py", line 131, in query_generator
    query_context = offline_utils.get_feature_view_query_context(
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast/infra/offline_stores/offline_utils.py", line 145, in get_feature_view_query_context
    table_subquery=feature_view.input.get_table_query_string(),
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast/data_source.py", line 419, in get_table_query_string
    raise NotImplementedError
NotImplementedError

Seems the method to.df() was not implemented in that version. Is that right?
How could be the right way to get my data as a DF?

I've also tried with feast 0.15.0
Got the same error.

Can you try reinstalling feast-postgres? The method is implemented https://github.com/nossrannug/feast-postgres/blob/main/feast_postgres/offline_stores/postgres_source.py#L87-L88

Thanks for the answer @nossrannug
I've tried, but get the same result.
Here are the steps I've made:

Create feature store with Postgres as registry and offline Store and Redis as Online store

project: xxxxx_repo
provider: local

registry:
    registry_store_type: feast_postgres.PostgreSQLRegistryStore
    path: feast_registry
    host: localhost
    port: 5432
    database: postgres
    db_schema: my_schema
    user: admin
    password: admin

offline_store:
    type: feast_postgres.PostgreSQLOfflineStore
    host: localhost
    port: 5432
    database: postgres
    db_schema: my_schema
    user: admin
    password: admin

online_store:
  type: redis
  connection_string: "localhost:6379,password=redis"

Setup a Docker compose with Redis and PostgreSQL with port and volumes mapped.
Create a NEW conda env with python 3.8

conda create -n featurestore python=3.8
conda activate featurestore

Install requirements.txt:

feast[redis]==0.16.0
feast-postgres==0.2.4

Ran feast apply

Registered entity Id
Registered feature view houses_features
Deploying infrastructure for houses_features

Try to get historical features using the code above:

#from feast.infra.offline_stores.file import SavedDatasetFileStorage
from feast import FeatureStore
import pandas as pd
from datetime import datetime

# repo_path points to the directory where the feature_store.yaml is stored
store = FeatureStore(repo_path=".")

entity_df = pd.DataFrame.from_dict(
    {
        "Id": [893, 414, 523],
        "event_timestamp": datetime(2022, 2, 10, 15, 28, 28),
    })

job = store.get_historical_features(
    entity_df=entity_df,
    features=["houses_features:YrSold", "houses_features:SaleCondition", "houses_features:SalePrice"],
)

job.to_df()

Got the error:

Traceback (most recent call last):
  File "house_test.py", line 21, in <module>
    job.to_df()
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast/infra/offline_stores/offline_store.py", line 43, in to_df
    features_df = self._to_df_internal()
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast_postgres/offline_stores/postgres.py", line 204, in _to_df_internal
    return self._to_arrow_internal().to_pandas()
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast_postgres/offline_stores/postgres.py", line 211, in _to_arrow_internal
    with self._query_generator() as query:
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast_postgres/offline_stores/postgres.py", line 131, in query_generator
    query_context = offline_utils.get_feature_view_query_context(
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast/infra/offline_stores/offline_utils.py", line 145, in get_feature_view_query_context
    table_subquery=feature_view.input.get_table_query_string(),
  File "/home/ubuntu/miniconda/envs/featurestore/lib/python3.8/site-packages/feast/data_source.py", line 419, in get_table_query_string
    raise NotImplementedError
NotImplementedError

Am I missing something?
Any ideas?

It seems to be an issue with your setup. The last file mentioned in the stack trace shouldn't be called. It should be calling this method: https://github.com/nossrannug/feast-postgres/blob/main/feast_postgres/offline_stores/postgres_source.py#L87 in this postgres_source file.
Can you create a small repo with code and steps to reproduce the issue?

Thanks again @nossrannug
Here is the repo: https://github.com/cpatrickalves/fsissue

The data source needs to match the offline store.
You need to change your offline_store in the yaml to file to use the FileSource. Or change the file source to be a PostgreSQLSource
# defines the feature source location
house_data_batch = FileSource(
    path="data/raw/houses.parquet",
    event_timestamp_column="created",
)

The data source needs to match the offline store.
I did not know that.
I thought It will load the parquet file in the SQL DB (offline store).

change the file source to be a PostgreSQLSource
In this case, do I need to load all data in some SQL database?  I have parquet files with + 10GB of data =(

Feast connects to an existing data source and treats that as the offline store. So you could have the parquet be the offline store. To use postgres offline store you'll need to get all the data into postgres first.
But you could have the parquet as the offline store and redis as online store and call feast materialize to populate redis with data from the file.

I got now.
I've miss understood offline stores.
My point to using PostgreSQL is that I have a multiuser environment, so sharing parquet files would be difficult.
Having all the data in an online DB would be better.
I've checked the script you have to load parquet files in DB (copy_parquet_to_postgres.py).
You have a con.execute("DROP TABLE IF EXISTS " + table) on it.
Wouldn't this be a bad practice as you will lose all historical data?
Thanks @nossrannug
I really appreciate your help.

Yes. This script is not meant for production. This is simply to copy the parquet that comes with the feast example to postgres so that feast-postgres can be tested. but you might be able to use it, with some modifications, to move your parquet to postgres as a one time thing.

I'll go ahead and close this issue. If there is anything else you can open a new issue or ping me on the Feast slack.

Everything work as expected now.
Thanks!


Unable to build profile images of wso2am latest version (2.0.0)

I'm trying to build separated profiles to wso2am latest version (2.0.0) but it only creates the default one. If I use version 1.10.0 it works well. Am I missing something or it just don't work with wso2am latest version?

@memorais Are you using Puppet to configure the Docker images? If so which version are you using? Make sure Puppet has the related Hieradata and the templates matching the product version you need to build.

@chamilad No, I'm not using puppet. I'm just putting the files in common/provision/default/files directory and using the build.sh script.

Profile images cannot be built with default provisioning scheme, would need to use Puppet. Please see the documentation for more information:
https://docs.wso2.com/display/DF120/WSO2+Dockerfiles+Documentation


Pretty terminal commands

Just added '$' in the beginning of terminal commands
I think it looks better. The project is amazing!

hey thanks for the idea. However we'd prefer not to add $ as it makes it hard for other OS like Windows to copy paste the command


环境 php7.3.1 swoole4.2.13 easyswoole3.1.14 报swFactoryProcess_finish (ERROR 1005): connection[fd=0]

[2019-02-14 02:29:32 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:32 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:32 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:32 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.
[2019-02-14 02:29:33 *11435.2]  NOTICE  swFactoryProcess_finish (ERROR 1005): connection[fd=0] does not exists.

这个我也遇到过，不过swoole官方 文档也有说明这个错误，但是没解释为什么，这个是不是在并发情况下才会出现的吧？

目前是swoole底层问题，请等待swoole升级


Increase test coverage in component_build_controller_pac.go and compo…

…nent_build_controller_simple_build.go
Properly handle errors when:
gitprovider can't be detected
pac secret doesn't exist
fails to generate pipeline run for simple build
When UndoPaCProvisionForComponent fails with persistent error set buildStatus.State to error
Added 1 sec sleep in AfterEach after component deletion, so it waits for pruner

Added 1 sec sleep in AfterEach after component deletion, so it waits for pruner

What would happen if not wait for the pruner?

Added 1 sec sleep in AfterEach after component deletion, so it waits for pruner

What would happen if not wait for the pruner?

tests are randomly failing due that race

Added 1 sec sleep in AfterEach after component deletion, so it waits for pruner

What would happen if not wait for the pruner?

well problem is that we have pruner controller active, so when you delete component, it triggers ALSO pruner, at that time already runs new test which creates component
and now there can be race between pruner cleaning OLD/NEW pipeline runs

/retest

/retest


Add Java SDK-based example to the Getting Started samples

This adds two sample Java applications to the Getting Started folder, mirroring what is already there in the Python subfolder. They're intended to be simple examples illustrating how to use the SDK, and to complement the Getting Started with the API article on the Help Center.
Might need some 'polishing' and also an improvement to the Maven config (new target?) to make it easier to run the code.

@rmalyona What's mainstream now: maven vs gradle? I think we should use what is in trend now because this is doc for at least the next 5+ years.

@rmalyona What's mainstream now: maven vs gradle? I think we should use what is in trend now because this is doc for at least the next 5+ years.

There's no strict winner here. Or team uses gradle, but I know other teams use Maven as well
I've latest developer survey, but it's dated 2019


GFCI transformer P/N not specified

I couldn't see where you specify a part number for the transformer. Do you have a recommended part from AliExpress, eBay, etc.
Thanks!

Hello. Hopefully you are getting answers.
I'm not sure why I am getting the messages in my inbox. Maybe I follow this
project on GitHub but I don't know the details of it.
On Wed, 22 Mar 2023, 15:52 Dr. George Smart, @.***>
wrote:

I couldn't see where you specify a part number for the transformer. Do you
have a recommended part from AliExpress, eBay, etc.
Thanks!
—
Reply to this email directly, view it on GitHub
https://github.com/fotherja/EV_Charger/issues/4, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ATAH6HSJXCCJEW5VG7CW6CDW5MN3LANCNFSM6AAAAAAWEANYYI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Hello, yes, I got the GFCI from the EVSE store. Here's the link: https://store.openevse.com/collections/all-products/products/gfci-current-transformer


Tutorial on custom IP cores in the FPGA

This PR does 3 things:

Create an Advanced Tutorials section after the existing Tutorials section
Move the Custom Control Rate tutorial to the advanced section
Add a new advanced tutorial about IP cores

@codecubepi if you get a chance, can you review this PR (see PDF above)?
I am curious if you think everything is explained clearly, things are missing, etc.
In other words, if you only had this tutorial to go by when you started with the FPGA work on the AMDC, would it be enough for you to understand how IP cores work?

Once @codecubepi reviews, I think we should merge this PR.
I have not implemented this in hardware yet, but let's have the folks starting with the AMDC and FPGA development this summer do this tutorial and gather the experimental results :)

Oh, one more thing I forgot to note above... would it be good to mention somewhere that the custom IP we create is able to be "Edited in IP Packager", but the IP provided by Vivado/etc is not? I know this was a question I asked you once.
I had it as a review note in my phone when I started my review the other night, but I didn't have it specifically attached to a certain section of the doc.

Thanks @codecubepi !
I think I addressed all your feedback, please see the latest commit for a diff.
Also, here is the updated tutorial in PDF form:
Tutorial Custom FPGA IP Core - AMDC Platform - v2.pdf
...this is quite the article! long! we'll have to see how this fares "in the wild" as new people read and complete the steps.

Looks great!


Routine Bag says that it only operates on a Mix or a Setty when it operates on Seqs and other things as well.

https://docs.perl6.org/routine/Bag
'hasfjdsfhdsjfhdsjakf'.comb.Bag # works fine and is Seq type before .Bag

The docs are correct but incomplete, fixed with 4afe7dd9b5.