[Bug]Each training period: ConcatOp:Dimension of inputs should match

Description

I'm a novice of deep learning. The training set prepared for 2stems.Sample_rate of training set is 8KHz,and the format of which is wav.
Each training period,Some errors were shown on the console that ConcatOp:Dimension of inputs should match:shape[0]=[,,2] vs. shape[1]=[,,1].
Details as follows:
Step to reproduce

spleeter train --verbose -p path of json path of audio training set

Output
gs\atmb\train\118.80T_1001_0015_Cessna_CockpitVoice2\vocals.wav' from 11.260
000000000002 to 31.26
INFO:spleeter:Loading audio b'K:\Anaconda3\Lib\site-packages\spleeter\confi
gs\atmb\train\118.80T_1001_0015_Cessna_CockpitVoice1\vocals.wav' from 11.260
000000000002 to 31.26
WARNING:tensorflow:Entity <bound method Node.stream of input(filename='K:\Anaco
nda3\Lib\site-packages\spleeter\configs\atmb\train\118.80T_1001_0015_Cess
na_CockpitVoice1\vocals.wav') > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bu
g, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attac
h the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of output(ar=44100, filename
='pipe:', format='f32le', ss='0:00:00.500000', t='0:00:20.000000') <152313499f09

could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, expor t AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of input(filename='K:\Anaco
nda3\Lib\site-packages\spleeter\configs\atmb\train\118.80T_1001_0015_Cess
na_CockpitVoice2\vocals.wav') <175cf1ae1684>> could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bu
g, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attac
h the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of input(filename='K:\Anaco
nda3\Lib\site-packages\spleeter\configs\atmb\train\118.80T_1001_0015_Cess
na_CockpitVoice2\vocals.wav') <175cf1ae1684>> could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bu
g, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attac
h the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of output(ar=44100, filename
='pipe:', format='f32le', ss='0:00:11.260000', t='0:00:20.000000') <f4673ba3c59a
could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, expor t AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of output(ar=44100, filename
='pipe:', format='f32le', ss='0:00:00.500000', t='0:00:20.000000') <94e14c0fda78
could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, expor t AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of input(filename='K:\Anaco
nda3\Lib\site-packages\spleeter\configs\atmb\train\118.80T_1001_0015_Cess
na_CockpitVoice1\vocals.wav') > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bu
g, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attac
h the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of output(ar=44100, filename
='pipe:', format='f32le', ss='0:00:11.260000', t='0:00:20.000000') <1b2581ce8eaa
could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, expor t AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Loading audio b'K:\Anaconda3\Lib\site-packages\spleeter\confi
gs\atmb\train\118.80T_1001_0015_Cessna_CockpitVoice1\other.wav' from 0.5 to
20.5
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Loading audio b'K:\Anaconda3\Lib\site-packages\spleeter\confi
gs\atmb\train\118.80T_1001_0015_Cessna_CockpitVoice2\other.wav' from 0.5 to
20.5
INFO:spleeter:Loading audio b'K:\Anaconda3\Lib\site-packages\spleeter\confi
gs\atmb\train\118.80T_1001_0015_Cessna_CockpitVoice2\other.wav' from 11.2600
00000000002 to 31.26
INFO:spleeter:Loading audio b'K:\Anaconda3\Lib\site-packages\spleeter\confi
gs\atmb\train\118.80T_1001_0015_Cessna_CockpitVoice1\other.wav' from 11.2600
00000000002 to 31.26
WARNING:tensorflow:Entity <bound method Node.stream of input(filename='K:\Anaco
nda3\Lib\site-packages\spleeter\configs\atmb\train\118.80T_1001_0015_Cess
na_CockpitVoice1\other.wav') <7b99e0a559c3>> could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bug
, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach
the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of output(ar=44100, filename
='pipe:', format='f32le', ss='0:00:00.500000', t='0:00:20.000000') <103881ea98f4
could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, expor t AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of input(filename='K:\Anaco
nda3\Lib\site-packages\spleeter\configs\atmb\train\118.80T_1001_0015_Cess
na_CockpitVoice2\other.wav') <937f23ce070b>> could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bug
, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach
the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of input(filename='K:\Anaco
nda3\Lib\site-packages\spleeter\configs\atmb\train\118.80T_1001_0015_Cess
na_CockpitVoice2\other.wav') <937f23ce070b>> could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bug
, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach
the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of output(ar=44100, filename
='pipe:', format='f32le', ss='0:00:11.260000', t='0:00:20.000000') <18ef7ee44f04
could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, expor t AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of output(ar=44100, filename
='pipe:', format='f32le', ss='0:00:00.500000', t='0:00:20.000000') <18fdbe3dd88a
could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, expor t AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
INFO:spleeter:Audio data loaded successfully
WARNING:tensorflow:Entity <bound method Node.stream of input(filename='K:\Anaco
nda3\Lib\site-packages\spleeter\configs\atmb\train\118.80T_1001_0015_Cess
na_CockpitVoice1\other.wav') <7b99e0a559c3>> could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bug
, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach
the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity <bound method Node.stream of output(ar=44100, filename
='pipe:', format='f32le', ss='0:00:11.260000', t='0:00:20.000000') <180187c01b07
could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, expor t AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Audio data loaded successfully
Traceback (most recent call last):
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1365, in _do_call
return fn(*args)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1350, in _run_fn
target_list, run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __
inference_Dataset_map_DatasetBuilder.random_time_crop_1250}} ConcatOp : Dimensio
ns of inputs should match: shape[0] = [858,1024,2] vs. shape[1] = [858,1024,1]
[[{{node concat}}]]
[[IteratorGetNext]]

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "K:\Anaconda3\Scripts\spleeter-script.py", line 10, in
sys.exit(entrypoint())
File "K:\Anaconda3\lib\site-packages\spleeter_main_.py", line 54, in entryp
oint
main(sys.argv)
File "K:\Anaconda3\lib\site-packages\spleeter_main_.py", line 46, in main
entrypoint(arguments, params)
File "K:\Anaconda3\lib\site-packages\spleeter\commands\train.py", line 98, in
entrypoint
evaluation_spec)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\tra
ining.py", line 473, in train_and_evaluate
return executor.run()
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\tra
ining.py", line 613, in run
return self.run_local()
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\tra
ining.py", line 714, in run_local
saving_listeners=saving_listeners)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\est
imator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\est
imator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\est
imator.py", line 1195, in _train_model_default
saving_listeners)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\est
imator.py", line 1494, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 754, in run
run_metadata=run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1259, in run
run_metadata=run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1360, in run
raise six.reraise(*original_exc_info)
File "K:\Anaconda3\lib\site-packages\six.py", line 693, in reraise
raise value
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1418, in run
run_metadata=run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1176, in run
return self._sess.run(*args, **kwargs)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 956, in run
run_metadata_ptr)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1359, in _do_run
run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  ConcatOp : Dimens
ions of inputs should match: shape[0] = [858,1024,2] vs. shape[1] = [858,1024,1]
     [[node concat (defined at \lib\site-packages\tensorflow_core\python\fra

mework\ops.py:1748) ]]
[[IteratorGetNext]]
Original stack trace for 'concat':
File "\Scripts\spleeter-script.py", line 10, in
sys.exit(entrypoint())
File "\lib\site-packages\spleeter_main_.py", line 54, in entrypoint
main(sys.argv)
File "\lib\site-packages\spleeter_main_.py", line 46, in main
entrypoint(arguments, params)
File "\lib\site-packages\spleeter\commands\train.py", line 98, in entrypoint
evaluation_spec)
File "\lib\site-packages\tensorflow_estimator\python\estimator\training.py", l
ine 473, in train_and_evaluate
return executor.run()
File "\lib\site-packages\tensorflow_estimator\python\estimator\training.py", l
ine 613, in run
return self.run_local()
File "\lib\site-packages\tensorflow_estimator\python\estimator\training.py", l
ine 714, in run_local
saving_listeners=saving_listeners)
File "\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py",
line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py",
line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py",
line 1195, in _train_model_default
saving_listeners)
File "\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py",
line 1490, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 1014, in init
stop_grace_period_secs=stop_grace_period_secs)
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 725, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 1207, in init
_WrappedSession.init(self, self._create_session())
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 1212, in _create_session
return self._sess_creator.create_session()
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 878, in create_session
self.tf_sess = self._session_creator.create_session()
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 638, in create_session
self._scaffold.finalize()
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 205, in finalize
default_ready_op)
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 287, in get_or_default
op = default_constructor()
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 201, in default_ready_op
], 0)
File "\lib\site-packages\tensorflow_core\python\util\dispatch.py", line 180, i
n wrapper
return target(*args, **kwargs)
File "\lib\site-packages\tensorflow_core\python\ops\array_ops.py", line 1420,
in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "\lib\site-packages\tensorflow_core\python\ops\gen_array_ops.py", line 12
56, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "\lib\site-packages\tensorflow_core\python\framework\op_def_library.py",
line 794, in _apply_op_helper
op_def=op_def)
File "\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507
, in new_func
return func(*args, **kwargs)
File "\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357,
in create_op
attrs, op_def, compute_device)
File "\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426,
in _create_op_internal
op_def=op_def)
File "\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748,
in init
self._traceback = tf_stack.extract_stack()
Environment

| OS                | Windows 7 |
| Installation type | Conda |
| Hardware spec     | CPU |
| spleeter 1.5.3  |
| tensorflow 1.15.0 |
Additional context

Same issue here. Any workaround?

Same issue here. Any workaround?

Found the reason, it's about the data preprocessing in my case. The audio file should all keep stereo instead of mono.


iui-text-block has no predefined color

Describe the bug (actual behavior)
<Text>Lorem Ipsum</Text> has no predefined text color it uses inherited one.
Output:

Lorem Ipsum

Computed color:

Result:

@veekeys might add additional info if needed.

I would expect that the inherited color is correct in most cases. If we were to set a color, it would likely be var(--iui-text-color).
I'm curious how you got 255, 255, 255, 0.85 on light theme. It is a dark theme value.

This example is from header component and we use 2 themes for it: dark for the header itself, and light for all dropdowns.
.portal-header {
  //using dark theme colors for all header elements
  @include theme-variables('dark');

  .iui-menu {
    //based on mockups dropdown should be in light theme
    @include theme-variables('light');
  }
}

And in this case color of text in light theme dropdown is inherited from the header that is in a dark theme.

we use 2 themes for it: dark for the header itself, and light for all dropdowns.

That is some weird usage, but ok lets see what we can do.
I wonder if this is a scoping issue with the CSS variables. If you manually set the color on your side, does it work?
.iui-text-block {
  color: var(--iui-text-color);
}


[Improve][Connector-V2][DingTalk] Unified exception for DingTalk sink connector

sub-task of #2930

Assign this to me, please.


Multiplayer Server crashes when Immersive Portals is installed (Server Tick 60+)

Hello!
so I've been trying to install this mod on my minecraft server, but every single time, and ONLY with this mod, my server gives me the error saying that the max server tick has gone over 60.00s and has to close... and I tried the technic of changing the max server tick to -1, and yes, the server does stay open, but everything is frozen, no mobs are moving, etc
am I doing something wrong or is this mod just not suitable for servers?
thx

Are you using Fabric 1.16.4? And show the crash report.

Are you using Fabric 1.16.4? And show the crash report.

I'm actually using the Forge version, for 1.16.4 and yes here is the crash report
crash-2020-09-25_23.45.09-server.txt

I'm actually using the Forge version, for 1.16.4 and yes here is the crash report
crash-2020-09-25_23.45.09-server.txt

Try to upgrade to 1.16.4

Try to upgrade to 1.16.4


How can an image be input into a model to output its scene graph information and bounding box information for visualization?

First of all, the work is very instructive, thank you! As a novice to scene graph generation, my current research requires methods to extract information from images using scene graph generation. So I'm curious about how to use the output of the model to generate a scene graph, and I'm confused about how to visualize the output. Any guidance would be greatly appreciated.

Hi :). I was also deeply impressed by the EGTR paper. I am facing the same issue. As I am not very knowledgeable in the SGG domain, I understand that it involves predicting relationships between objects. I need to perform the SGG as a preliminary step to use SG embedding for downstream tasks. I would like to refer to EGTR as the SGG module, but could you please let me know how the output scene graph is generated?

First of all, I have not attempted to extract the scene graph well other than measuring measures for evaluation.
So, although I share the code snippet for extracting the scene graph, I recommend that you improve it to suit your own needs.
Following code snippet is based on evaluate_egtr.py.
from glob import glob

import torch
from PIL import Image

from model.deformable_detr import DeformableDetrConfig, DeformableDetrFeatureExtractor
from model.egtr import DetrForSceneGraphGeneration

# config
architecture = "SenseTime/deformable-detr"
min_size = 800
max_size = 1333
artifact_path = YOUR_ARTIFACT_PATH

# feature extractor
feature_extractor = DeformableDetrFeatureExtractor.from_pretrained(
    architecture, size=min_size, max_size=max_size
)

# inference image
image = Image.open(YOUR_IMAGE_PATH)
image = feature_extractor(image, return_tensors="pt")

# model
config = DeformableDetrConfig.from_pretrained(artifact_path)
model = DetrForSceneGraphGeneration.from_pretrained(
    architecture, config=config, ignore_mismatched_sizes=True
)
ckpt_path = sorted(
    glob(f"{artifact_path}/checkpoints/epoch=*.ckpt"),
    key=lambda x: int(x.split("epoch=")[1].split("-")[0]),
)[-1]
state_dict = torch.load(ckpt_path, map_location="cpu")["state_dict"]
for k in list(state_dict.keys()):
    state_dict[k[6:]] = state_dict.pop(k)  # "model."

model.load_state_dict(state_dict)
model.cuda()
model.eval()

# output
outputs = model(
    pixel_values=image['pixel_values'].cuda(),
    pixel_mask=image['pixel_mask'].cuda(),
    output_attention_states=True
)

pred_logits = outputs['logits'][0]
obj_scores, pred_classes = torch.max(pred_logits.softmax(-1), -1)
pred_boxes = outputs['pred_boxes'][0]

pred_connectivity = outputs['pred_connectivity'][0]
pred_rel = outputs['pred_rel'][0]
pred_rel = torch.mul(pred_rel, pred_connectivity)

# get valid objects and triplets
obj_threshold = YOUR_OBJ_THRESHOLD
valid_obj_indices = (obj_scores >= obj_threshold).nonzero()[:, 0]

valid_obj_classes = pred_classes[valid_obj_indices] # [num_valid_objects]
valid_obj_boxes = pred_boxes[valid_obj_indices] # [num_valid_objects, 4]

rel_threshold = YOUR_REL_THRESHOLD
valid_triplets = (pred_rel[valid_obj_indices][:, valid_obj_indices] >= rel_threshold).nonzero() # [num_valid_triplets, 3]

You can generate a scene graph based on valid_obj_classes, valid_obj_boxes, and valid_triplets.

valid_obj_classes: object classes
valid_obj_boxes: object bounding boxes (cxcywh format)
valid_triplets: relation triplets (subject entity index, object entity index, relation class)

Please note that subject entity index and object entity index indicate the indices of valid objects.

I built a scene graph using thresholds in this example, but it can also be implemented by selecting the top k objects or triplets.
Since the thresholds have never been explored, it may be important to set the threshold well.

@jinbae Thank you for providing the reference template, which has been very helpful to me. However, when I loaded the provided model weights according to the readme.md file and input images from the VG database for inference, I found that the inference results were very poor. I followed the method you provided, activated the logits through softmax, and obtained the maximum classification probability. When I checked the specific values, they were all very low, with the highest being around 0.5, which is clearly problematic. Could you please tell me where the problem lies?

@PiPiSang
(1) obj boxes

valid_obj_boxes: object bounding boxes (cxcywh format)

As I mentioned before, pred_boxes are cxcywh format.
Please make sure that pred_boxes have been converted to xyxy format before bbox visualization.
(2) obj scores
We used Deformable DETR rather than DETR, and in Deformable DETR, focal loss is used for object detection instead of cross entropy loss.

obj_scores, pred_classes = torch.max(pred_logits.softmax(-1), -1)

Therefore, It is more natural to use sigmoid instead of softmax (https://github.com/huggingface/transformers/blob/409fcfdfccde77a14b7cc36972b774cabc371ae1/src/transformers/models/deformable_detr/image_processing_deformable_detr.py#L1555), but we used softmax to get obj_scores.
Obj_scores may be low compared to models trained with the cross entropy loss.

@jinbae
Thank you very much for your response regarding scene graph visualization. I followed the code you provided for inference on a single image and used the pretrained weights for the oi dataset as mentioned in the README for inference. However, I encountered the following error.

I then tried using the weights I trained myself on the oi dataset, but the same error occurred. I would like to know why the class dimension in the weights is 91 instead of 601, which is the number of classes in the oi dataset. I am eagerly awaiting your response.

@Aoihigashi In fact, this is not an error. This so-called error is emitted during the execution of the model initialization. You can try debugging this test code, and you will find that after executing the code on line 229, the console will output this message. Moreover, the code on line 229 does not actually load the trained weights into the model. This code simply creates a model that matches the structure specified in the configuration file and initializes it with the official default weights. Therefore, this message is indicating that the structure of the model does not fully match the official model structure, which leads to the inability to completely load the weights during initialization. The code that truly loads our own trained weights into the model is on line 240. So, you can relax, there is no issue.

@PiPiSang
Thank you! I see, that’s exactly the case, and it does run. However, I have to set the obj_threshold and rel_threshold very low (0.1) to get any output. Have you encountered a similar issue?

@Aoihigashi Yes. The obj_threshold and rel_threshold for me are 0.3 and 1e-4. But based on my tests, I strongly suggest that you directly select the top_k triplets based on their scores. Then, you can read the function where the author calculates the loss in train_egtr.py. This demonstrates how to select the top_k triplets and perform the transformation.

@PiPiSang
Thank you for your suggestions and response. I will try following the advice.

@Aoihigashi Yes. The obj_threshold and rel_threshold for me are 0.3 and 1e-4. But based on my tests, I strongly suggest that you directly select the top_k triplets based on their scores. Then, you can read the function where the author calculates the loss in train_egtr.py. This demonstrates how to select the top_k triplets and perform the transformation.

Could you please explain in detail how to operate it? Thank you very much.

@riverjunhao Of course, here’s my code. However, I have only visualized the bounding boxes. The scene graph is stored in a text file in a textual format. Since I am not working on scene graph research, I haven’t delved into how to visualize the scene graph on an image. So, if needed, you can explore how to implement this feature yourself. If you could share your findings, I believe it would be very helpful for other beginners.
`
EGTR
Copyright (c) 2024-present NAVER Cloud Corp.
Apache-2.0
import os
os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"
import argparse
import json
from glob import glob
from PIL import Image, ImageDraw, ImageFont
import torch
torch.cuda.set_device(2)
from lib.pytorch_misc import argsort_desc
from torch.utils.data import DataLoader
from tqdm import tqdm
from data.open_image import OIDataset
from data.visual_genome import VGDataset
from lib.evaluation.coco_eval import CocoEvaluator
from lib.evaluation.oi_eval import OIEvaluator
from lib.evaluation.sg_eval import (
BasicSceneGraphEvaluator,
calculate_mR_from_evaluator_list,
)
from model.deformable_detr import DeformableDetrConfig, DeformableDetrFeatureExtractor
from model.egtr import DetrForSceneGraphGeneration
from train_egtr import collate_fn, evaluate_batch
import numpy as np
@torch.no_grad()
def calculate_fps(model, dataloader, top_k):
model.eval()
i = 0
for batch in tqdm(dataloader):
i = i + 1
if i > 10:
break
outputs = model(
pixel_values=batch["pixel_values"].cuda(),
pixel_mask=batch["pixel_mask"].cuda(),
output_attentions=False,
output_attention_states=True,
output_hidden_states=True,
)
pred_logits = outputs['logits'][0]
obj_scores, pred_classes = torch.max(pred_logits.softmax(-1), -1)
pred_classes_numpy = pred_classes.cpu().clone().numpy()
sub_ob_scores = torch.outer(obj_scores, obj_scores)
sub_ob_scores[
torch.arange(pred_logits.size(0)), torch.arange(pred_logits.size(0))
] = 0.0  # prevent self-connection
    pred_boxes = outputs['pred_boxes'][0]
    pred_boxes_numpy = pred_boxes.cpu().clone().numpy()
    pred_connectivity = outputs["pred_connectivity"][0]
    pred_rel = outputs["pred_rel"][0]
    pred_rel = torch.mul(pred_rel, pred_connectivity)
    triplet_scores = torch.mul(pred_rel.max(-1)[0], sub_ob_scores)
    pred_rel_inds = argsort_desc(triplet_scores.cpu().clone().numpy())[:top_k, :]  # [pred_rels, 2(s,o)]
    rel_scores = (pred_rel.cpu().clone().numpy()[pred_rel_inds[:, 0], pred_rel_inds[:, 1]])
    pred_rels = np.column_stack((pred_rel_inds, rel_scores.argmax(1)))
    triplets, triplets_box = get_triplets(pred_rels, pred_classes_numpy, pred_boxes_numpy)

    # get valid objects and triplets
    obj_threshold = 0.3
    # 提取出概率大于阈值的对象的索引
    valid_obj_indices = (obj_scores >= obj_threshold).nonzero()[:top_k, 0]
    valid_obj_classes = pred_classes[valid_obj_indices] # [num_valid_objects]
    valid_obj_boxes = pred_boxes[valid_obj_indices] # [num_valid_objects, 4](x_center, y_center, width, height)

    # rel_threshold = 1e-4
    # valid_triplets = (pred_rel[valid_obj_indices][:, valid_obj_indices] >= rel_threshold).nonzero() # [num_valid_triplets, 3](subject, object, relation class)
    # valid_triplets = valid_triplets.cpu().numpy()
    # triplets = []
    # for sub, obj, rel in valid_triplets:
    #     try:
    #         sub = pred_classes[sub].item()
    #         obj = pred_classes[sub].item()
    #         triplets.append([sub, obj, rel])
    #     except:
    #         continue

    if not os.path.exists("./val_images"):
        os.mkdir("./val_images")
    if not os.path.exists('./relationship'):
        os.mkdir("./relationship")
    if not os.path.exists(f"./val_images/{batch['id']}.png"):
        image = Image.fromarray(batch['img'], 'RGB')
        filename = f"./val_images/{batch['id']}"
        image.save(f"{filename}.png")
        visualization(image, valid_obj_boxes, filename, valid_obj_classes)
        write_triplets(triplets, batch['id'])

def get_triplets(pred_triplets, classes, boxes):
triplets, triplet_boxes = [], []
for sub, obj, rel in pred_triplets:
triplet_boxes.append([boxes[sub], boxes[obj]])
sub = id2label[classes[sub]]
obj = id2label[classes[obj]]
rel = id2relation[rel]
triplets.append([obj, rel, sub])
return triplets, triplet_boxes
def write_triplets(triplets, id):
file_path = f'./relationship/{id}.txt'
with open(file_path, 'w') as file:
for sub, rel, obj in triplets:
file.write(str(obj) + ' ' + str(rel) + ' ' + str(sub) + '\n')
def get_relation(triplets, id):
relation_list = []
for sub, obj, rel in triplets:
sub = id2label[sub]
obj = id2label[obj]
rel = id2relation[rel]
relation_list.append([sub, obj, rel])
file_path = f'./relationship/{id}.txt'
with open(file_path, 'w') as file:
for sub, obj, rel in relation_list:
file.write(str(obj) + ' ' + str(rel) + ' ' + str(sub) + '\n')
def visualization(img, box, filename, obj_class):
colors = ['red', 'blue', 'orange', 'green', 'yellow', 'purple']
font = ImageFont.truetype("/usr/share/fonts/truetype/tlwg/Garuda.ttf", size=100)
obj_list = obj_class.cpu().numpy()
bboxes = box.cpu().numpy()
draw = ImageDraw.Draw(img)
font = ImageFont.load_default()
i = 0
for bbox, obj in zip(bboxes, obj_list):
x_center, y_center, width, height = bbox
x_max = int((x_center + width/2) * img.width)
y_max = int((y_center + height/2) * img.height)
x_min = int((x_center - width/2) * img.width)
y_min = int((y_center - height/2) * img.height)
obj = id2label[obj]
text_width, text_height = 10, 10
text_x = max(x_min + text_width + 5, 0)
text_y = y_min + text_height + 5
draw.text((text_x, text_y), str(obj), fill=colors[i], font=font)
draw.rectangle([(x_min, y_min), (x_max, y_max)], outline=colors[i], fill=None, width=3)
i = (i + 1)%6
img.save(f"{filename}_box.png")
Reference: https://github.com/facebookresearch/detr/blob/main/engine.py
@torch.no_grad()
def evaluate(
model,
dataloader,
num_labels,
multiple_sgg_evaluator=None,
single_sgg_evaluator=None,
oi_evaluator=None,
coco_evaluator=None,
feature_extractor=None,
):
metric_dict = {}
model.eval()
multiple_sgg_evaluator_list = []  # mR@k (for each rel category)
single_sgg_evaluator_list = []
if multiple_sgg_evaluator is not None:
    for index, name in enumerate(dataloader.dataset.rel_categories):
        multiple_sgg_evaluator_list.append(
            (index, name, BasicSceneGraphEvaluator.all_modes(multiple_preds=True))
        )
if single_sgg_evaluator is not None:
    for index, name in enumerate(dataloader.dataset.rel_categories):
        single_sgg_evaluator_list.append(
            (index, name, BasicSceneGraphEvaluator.all_modes(multiple_preds=False))
        )

for batch in tqdm(dataloader):
    outputs = model(
        pixel_values=batch["pixel_values"].cuda(),
        pixel_mask=batch["pixel_mask"].cuda(),
        output_attentions=False,
        output_attention_states=True,
        output_hidden_states=True,
    )
    targets = batch["labels"]
    evaluate_batch(
        outputs,
        targets,
        multiple_sgg_evaluator,
        multiple_sgg_evaluator_list,
        single_sgg_evaluator,
        single_sgg_evaluator_list,
        oi_evaluator,
        num_labels,
    )
    if coco_evaluator is not None:
        orig_target_sizes = torch.stack(
            [target["orig_size"] for target in targets], dim=0
        )
        results = feature_extractor.post_process(
            outputs, orig_target_sizes.to(model.device)
        )  # convert outputs of model to COCO api
        res = {
            target["image_id"].item(): output
            for target, output in zip(targets, results)
        }
        coco_evaluator.update(res)

if coco_evaluator is not None:
    coco_evaluator.synchronize_between_processes()
    coco_evaluator.accumulate()
    coco_evaluator.summarize()
    metric_dict.update({"AP50": coco_evaluator.coco_eval["bbox"].stats[1]})

if multiple_sgg_evaluator is not None:
    recall = multiple_sgg_evaluator["sgdet"].print_stats()
    mean_recall = calculate_mR_from_evaluator_list(
        multiple_sgg_evaluator_list, "sgdet", multiple_preds=True
    )
    metric_dict.update(recall)
    metric_dict.update(mean_recall)

if single_sgg_evaluator is not None:
    recall = single_sgg_evaluator["sgdet"].print_stats()
    mean_recall = calculate_mR_from_evaluator_list(
        single_sgg_evaluator_list, "sgdet", multiple_preds=False
    )
    recall = {f"(single){key}": value for key, value in recall.items()}
    mean_recall = {f"(single){key}": value for key, value in mean_recall.items()}
    metric_dict.update(recall)
    metric_dict.update(mean_recall)

if oi_evaluator is not None:
    metrics = oi_evaluator.aggregate_metrics()
    metric_dict.update(metrics)

return metric_dict

def evaluate_fn(batch, feature_extractor):
pixel_values = [item[0] for item in batch]
encoding = feature_extractor.pad_and_create_pixel_mask(
pixel_values, return_tensors="pt"
)
labels = [item[1] for item in batch]
img = batch[0][2]
id = batch[0][1]["id"]
batch = {}
batch["pixel_values"] = encoding["pixel_values"]
batch["pixel_mask"] = encoding["pixel_mask"]
batch["labels"] = labels
batch["img"] = img
batch["id"] = id
return batch
if name == "main":
def str2bool(v):
    if isinstance(v, bool):
        return v
    if v.lower() in ("yes", "true", "t", "y", "1"):
        return True
    elif v.lower() in ("no", "false", "f", "n", "0"):
        return False
    else:
        raise argparse.ArgumentTypeError("Boolean value expected.")

parser = argparse.ArgumentParser()
# Path
parser.add_argument("--data_path", type=str, default="dataset/visual_genome")
parser.add_argument(
    "--artifact_path",
    type=str,
    default="./egtr_VG",
    required=False,
)

# Architecture
parser.add_argument("--architecture", type=str, default="SenseTime/deformable-detr")
parser.add_argument("--num_queries", type=int, default=200)

# Evaluation
parser.add_argument("--split", type=str, default="test", choices=["val", "test"])
parser.add_argument("--eval_batch_size", type=int, default=1)
parser.add_argument("--eval_single_preds", type=str2bool, default=True)
parser.add_argument("--eval_multiple_preds", type=str2bool, default=False)

parser.add_argument("--logit_adjustment", type=str2bool, default=False)
parser.add_argument("--logit_adj_tau", type=float, default=0.3)

# FPS
parser.add_argument("--min_size", type=int, default=600)
parser.add_argument("--max_size", type=int, default=1000)
parser.add_argument("--infer_only", type=str2bool, default=True)
parser.add_argument("--top_k", type=int, default=10)

# Speed up
parser.add_argument("--num_workers", type=int, default=1)
args, unknown = parser.parse_known_args()  # to ignore args when training

# Feature extractor
feature_extractor = DeformableDetrFeatureExtractor.from_pretrained(
    args.architecture, size=args.min_size, max_size=args.max_size
)

# Dataset
if "visual_genome" in args.data_path:
    test_dataset = VGDataset(
        data_folder=args.data_path,
        feature_extractor=feature_extractor,
        split=args.split,
        num_object_queries=args.num_queries,
    )
    id2relation = test_dataset.rel_categories
    id2label = {
        k - 1: v["name"] for k, v in test_dataset.coco.cats.items()
    }  # 0 ~ 149
    coco_evaluator = CocoEvaluator(
        test_dataset.coco, ["bbox"]
    )  # initialize evaluator with ground truths
    oi_evaluator = None
elif "open-image" in args.data_path:
    test_dataset = OIDataset(
        data_folder=args.data_path,
        feature_extractor=feature_extractor,
        split=args.split,
        num_object_queries=args.num_queries,
    )
    id2label = test_dataset.classes_to_ind  # 0 ~ 600
    oi_evaluator = OIEvaluator(
        test_dataset.rel_categories, test_dataset.ind_to_classes
    )
    coco_evaluator = None

# Dataloader
test_dataloader = DataLoader(
    test_dataset,
    collate_fn=lambda x: evaluate_fn(x, feature_extractor),
    batch_size=args.eval_batch_size,
    pin_memory=True,
    num_workers=args.num_workers,
    persistent_workers=True,
)

# Evaluator
multiple_sgg_evaluator = None
single_sgg_evaluator = None
if args.eval_multiple_preds:
    multiple_sgg_evaluator = BasicSceneGraphEvaluator.all_modes(multiple_preds=True)
if args.eval_single_preds:
    single_sgg_evaluator = BasicSceneGraphEvaluator.all_modes(multiple_preds=False)

# Model
config = DeformableDetrConfig.from_pretrained(args.artifact_path)
config.logit_adjustment = args.logit_adjustment
config.logit_adj_tau = args.logit_adj_tau

model = DetrForSceneGraphGeneration.from_pretrained(
    args.architecture, config=config, ignore_mismatched_sizes=True
)
ckpt_path = sorted(
    glob(f"{args.artifact_path}/checkpoints/epoch=*.ckpt"),
    key=lambda x: int(x.split("epoch=")[1].split("-")[0]),
)[-1]
state_dict = torch.load(ckpt_path, map_location="cpu")["state_dict"]
for k in list(state_dict.keys()):
    state_dict[k[6:]] = state_dict.pop(k)  # "model."

model.load_state_dict(state_dict)
model.cuda()
model.eval()

# FPS
if args.infer_only:
    calculate_fps(model, test_dataloader, args.top_k)
# Eval
else:
    metric = evaluate(
        model,
        test_dataloader,
        max(id2label.keys()) + 1,
        multiple_sgg_evaluator,
        single_sgg_evaluator,
        oi_evaluator,
        coco_evaluator,
        feature_extractor,
    )

    # Save eval metric
    device = "".join(torch.cuda.get_device_name(0).split()[1:2])
    filename = f'{ckpt_path.replace(".ckpt", "")}__{args.split}__{len(test_dataloader)}__{device}'
    if args.logit_adjustment:
        filename += f"__la_{args.logit_adj_tau}"
    metric["eval_arg"] = args.__dict__
    with open(f"{filename}.json", "w") as f:
        json.dump(metric, f)
    print("metric is saved in", f"{filename}.json")

`

@riverjunhao Of course, here’s my code. However, I have only visualized the bounding boxes. The scene graph is stored in a text file in a textual format. Since I am not working on scene graph research, I haven’t delved into how to visualize the scene graph on an image. So, if needed, you can explore how to implement this feature yourself. If you could share your findings, I believe it would be very helpful for other beginners.
`@torch.no_grad()
def calculate_fps(model, dataloader, top_k):
model.eval()
i = 0
for batch in tqdm(dataloader):
i = i + 1
if i > 10:
break
outputs = model(
pixel_values=batch["pixel_values"].cuda(),
pixel_mask=batch["pixel_mask"].cuda(),
output_attentions=False,
output_attention_states=True,
output_hidden_states=True,
)
pred_logits = outputs['logits'][0]
obj_scores, pred_classes = torch.max(pred_logits.softmax(-1), -1)
pred_classes_numpy = pred_classes.cpu().clone().numpy()
sub_ob_scores = torch.outer(obj_scores, obj_scores)
sub_ob_scores[
torch.arange(pred_logits.size(0)), torch.arange(pred_logits.size(0))
] = 0.0  # prevent self-connection
    pred_boxes = outputs['pred_boxes'][0]
    pred_boxes_numpy = pred_boxes.cpu().clone().numpy()
    pred_connectivity = outputs["pred_connectivity"][0]
    pred_rel = outputs["pred_rel"][0]
    pred_rel = torch.mul(pred_rel, pred_connectivity)
    triplet_scores = torch.mul(pred_rel.max(-1)[0], sub_ob_scores)
    pred_rel_inds = argsort_desc(triplet_scores.cpu().clone().numpy())[:top_k, :]  # [pred_rels, 2(s,o)]
    rel_scores = (pred_rel.cpu().clone().numpy()[pred_rel_inds[:, 0], pred_rel_inds[:, 1]])
    pred_rels = np.column_stack((pred_rel_inds, rel_scores.argmax(1)))
    # 获取三元组中物体对应的编号，及每个三元组中物体的 bounding box
    triplets, triplets_box = get_triplets(pred_rels, pred_classes_numpy, pred_boxes_numpy)

    # get valid objects and triplets
    obj_threshold = 0.3
    valid_obj_indices = (obj_scores >= obj_threshold).nonzero()[:top_k, 0]
    valid_obj_classes = pred_classes[valid_obj_indices] # [num_valid_objects]
    valid_obj_boxes = pred_boxes[valid_obj_indices] # [num_valid_objects, 4](x_center, y_center, width, height)
    if not os.path.exists("./val_images"):
        os.mkdir("./val_images")
    if not os.path.exists('./relationship'):
        os.mkdir("./relationship")
    if not os.path.exists(f"./val_images/{batch['id']}.png"):
        image = Image.fromarray(batch['img'], 'RGB')
        filename = f"./val_images/{batch['id']}"
        image.save(f"{filename}.png")
        visualization(image, valid_obj_boxes, filename, valid_obj_classes)
        write_triplets(triplets, batch['id'])

def get_triplets(pred_triplets, classes, boxes):
triplets, triplet_boxes = [], []
for sub, obj, rel in pred_triplets:
triplet_boxes.append([boxes[sub], boxes[obj]])
sub = id2label[classes[sub]]
obj = id2label[classes[obj]]
rel = id2relation[rel]
triplets.append([obj, rel, sub])
return triplets, triplet_boxes
def write_triplets(triplets, id):
file_path = f'./relationship/{id}.txt'
with open(file_path, 'w') as file:
for sub, rel, obj in triplets:
file.write(str(obj) + ' ' + str(rel) + ' ' + str(sub) + '\n')
def get_relation(triplets, id):
relation_list = []
for sub, obj, rel in triplets:
sub = id2label[sub]
obj = id2label[obj]
rel = id2relation[rel]
relation_list.append([sub, obj, rel])
file_path = f'./relationship/{id}.txt'
with open(file_path, 'w') as file:
for sub, obj, rel in relation_list:
file.write(str(obj) + ' ' + str(rel) + ' ' + str(sub) + '\n')
def visualization(img, box, filename, obj_class):
colors = ['red', 'blue', 'orange', 'green', 'yellow', 'purple']
font = ImageFont.truetype("/usr/share/fonts/truetype/tlwg/Garuda.ttf", size=100)
obj_list = obj_class.cpu().numpy()
bboxes = box.cpu().numpy()
draw = ImageDraw.Draw(img)
font = ImageFont.load_default()
i = 0
for bbox, obj in zip(bboxes, obj_list):
x_center, y_center, width, height = bbox
x_max = int((x_center + width/2) * img.width)
y_max = int((y_center + height/2) * img.height)
x_min = int((x_center - width/2) * img.width)
y_min = int((y_center - height/2) * img.height)
obj = id2label[obj]
text_width, text_height = 10, 10
text_x = max(x_min + text_width + 5, 0)
text_y = y_min + text_height + 5
draw.text((text_x, text_y), str(obj), fill=colors[i], font=font)
draw.rectangle([(x_min, y_min), (x_max, y_max)], outline=colors[i], fill=None, width=3)
i = (i + 1)%6
img.save(f"{filename}_box.png")` I modified the calculate_fps function in the original file evaluate_egtr.py and added some necessary auxiliary functions. However, since I haven’t tried this code for more than a month, there might be some issues.

Here’s my code. However, I have only visualized the bounding boxes. The scene graph is stored in a text file in a textual format. Since I am not working on scene graph research, I haven’t delved into how to visualize the scene graph on an image. So, if needed, you can explore how to implement this feature yourself. If you could share your findings, I believe it would be very helpful for other beginners.
import os
import argparse
import json
from glob import glob
from PIL import Image, ImageDraw, ImageFont
import torch

@torch.no_grad()
def calculate_fps(model, dataloader, top_k):
    model.eval()
    i = 0
    for batch in tqdm(dataloader):
        i = i + 1
        if i > 10:
            break
        outputs = model(
            pixel_values=batch["pixel_values"].cuda(),
            pixel_mask=batch["pixel_mask"].cuda(),
            output_attentions=False,
            output_attention_states=True,
            output_hidden_states=True,
        )
        pred_logits = outputs['logits'][0]
        obj_scores, pred_classes = torch.max(pred_logits.softmax(-1), -1)
        pred_classes_numpy = pred_classes.cpu().clone().numpy()
        sub_ob_scores = torch.outer(obj_scores, obj_scores)
        sub_ob_scores[
            torch.arange(pred_logits.size(0)), torch.arange(pred_logits.size(0))
        ] = 0.0  # prevent self-connection

        pred_boxes = outputs['pred_boxes'][0]
        pred_boxes_numpy = pred_boxes.cpu().clone().numpy()
        pred_connectivity = outputs["pred_connectivity"][0]
        pred_rel = outputs["pred_rel"][0]
        pred_rel = torch.mul(pred_rel, pred_connectivity)
        triplet_scores = torch.mul(pred_rel.max(-1)[0], sub_ob_scores)
        pred_rel_inds = argsort_desc(triplet_scores.cpu().clone().numpy())[:top_k, :]  # [pred_rels, 2(s,o)]
        rel_scores = (pred_rel.cpu().clone().numpy()[pred_rel_inds[:, 0], pred_rel_inds[:, 1]])
        pred_rels = np.column_stack((pred_rel_inds, rel_scores.argmax(1)))
        triplets, triplets_box = get_triplets(pred_rels, pred_classes_numpy, pred_boxes_numpy)

        # Get valid objects and triplets
        obj_threshold = 0.3
        valid_obj_indices = (obj_scores >= obj_threshold).nonzero()[:top_k, 0]
        valid_obj_classes = pred_classes[valid_obj_indices]  # [num_valid_objects]
        valid_obj_boxes = pred_boxes[valid_obj_indices]  # [num_valid_objects, 4](x_center, y_center, width, height)

        if not os.path.exists("./val_images"):
            os.mkdir("./val_images")
        if not os.path.exists('./relationship'):
            os.mkdir("./relationship")
        if not os.path.exists(f"./val_images/{batch['id']}.png"):
            image = Image.fromarray(batch['img'], 'RGB')
            filename = f"./val_images/{batch['id']}"
            image.save(f"{filename}.png")
            visualization(image, valid_obj_boxes, filename, valid_obj_classes)
            write_triplets(triplets, batch['id'])

def get_triplets(pred_triplets, classes, boxes):
    triplets, triplet_boxes = [], []
    for sub, obj, rel in pred_triplets:
        triplet_boxes.append([boxes[sub], boxes[obj]])
        sub = id2label[classes[sub]]
        obj = id2label[classes[obj]]
        rel = id2relation[rel]
        triplets.append([obj, rel, sub])
    return triplets, triplet_boxes

def write_triplets(triplets, id):
    file_path = f'./relationship/{id}.txt'
    with open(file_path, 'w') as file:
        for sub, rel, obj in triplets:
            file.write(str(obj) + ' ' + str(rel) + ' ' + str(sub) + '\n')

def get_relation(triplets, id):
    relation_list = []
    for sub, obj, rel in triplets:
        sub = id2label[sub]
        obj = id2label[obj]
        rel = id2relation[rel]
        relation_list.append([sub, obj, rel])
    file_path = f'./relationship/{id}.txt'
    with open(file_path, 'w') as file:
        for sub, obj, rel in relation_list:
            file.write(str(obj) + ' ' + str(rel) + ' ' + str(sub) + '\n')

def visualization(img, box, filename, obj_class):
    colors = ['red', 'blue', 'orange', 'green', 'yellow', 'purple']
    font = ImageFont.truetype("/usr/share/fonts/truetype/tlwg/Garuda.ttf", size=100)
    obj_list = obj_class.cpu().numpy()
    bboxes = box.cpu().numpy()
    draw = ImageDraw.Draw(img)
    font = ImageFont.load_default()
    i = 0
    for bbox, obj in zip(bboxes, obj_list):
        x_center, y_center, width, height = bbox
        x_max = int((x_center + width/2) * img.width)
        y_max = int((y_center + height/2) * img.height)
        x_min = int((x_center - width/2) * img.width)
        y_min = int((y_center - height/2) * img.height)
        obj = id2label[obj]
        text_width, text_height = 10, 10
        text_x = max(x_min + text_width + 5, 0)
        text_y = y_min + text_height + 5
        draw.text((text_x, text_y), str(obj), fill=colors[i], font=font)
        draw.rectangle([(x_min, y_min), (x_max, y_max)], outline=colors[i], fill=None, width=3)
        i = (i + 1)%6
    img.save(f"{filename}_box.png")“””I modified the calculate_fps function in the original file evaluate_egtr.py and added some necessary auxiliary functions. However, since I haven’t tried this code for more than a month, there might be some issues.

@riverjunhao Sure! Here’s my code. However, I have only visualized the bounding boxes. The scene graph is stored in a text file in a textual format. Since I am not working on scene graph research, I haven’t delved into how to visualize the scene graph on an image. So, if needed, you can explore how to implement this feature yourself. If you could share your findings, I believe it would be very helpful for other beginners.I modified the calculate_fps function in the original file evaluate_egtr.py and added some necessary auxiliary functions. However, since I haven’t tried this code for more than a month, there might be some issues.
import os
import argparse
import json
from glob import glob
from PIL import Image, ImageDraw, ImageFont
import torch

@torch.no_grad()
def calculate_fps(model, dataloader, top_k):
    model.eval()
    i = 0
    for batch in tqdm(dataloader):
        i = i + 1
        if i > 10:
            break
        outputs = model(
            pixel_values=batch["pixel_values"].cuda(),
            pixel_mask=batch["pixel_mask"].cuda(),
            output_attentions=False,
            output_attention_states=True,
            output_hidden_states=True,
        )
        pred_logits = outputs['logits'][0]
        obj_scores, pred_classes = torch.max(pred_logits.softmax(-1), -1)
        pred_classes_numpy = pred_classes.cpu().clone().numpy()
        sub_ob_scores = torch.outer(obj_scores, obj_scores)
        sub_ob_scores[
            torch.arange(pred_logits.size(0)), torch.arange(pred_logits.size(0))
        ] = 0.0  # prevent self-connection

        pred_boxes = outputs['pred_boxes'][0]
        pred_boxes_numpy = pred_boxes.cpu().clone().numpy()
        pred_connectivity = outputs["pred_connectivity"][0]
        pred_rel = outputs["pred_rel"][0]
        pred_rel = torch.mul(pred_rel, pred_connectivity)
        triplet_scores = torch.mul(pred_rel.max(-1)[0], sub_ob_scores)
        pred_rel_inds = argsort_desc(triplet_scores.cpu().clone().numpy())[:top_k, :]  # [pred_rels, 2(s,o)]
        rel_scores = (pred_rel.cpu().clone().numpy()[pred_rel_inds[:, 0], pred_rel_inds[:, 1]])
        pred_rels = np.column_stack((pred_rel_inds, rel_scores.argmax(1)))
        triplets, triplets_box = get_triplets(pred_rels, pred_classes_numpy, pred_boxes_numpy)

        # Get valid objects and triplets
        obj_threshold = 0.3
        valid_obj_indices = (obj_scores >= obj_threshold).nonzero()[:top_k, 0]
        valid_obj_classes = pred_classes[valid_obj_indices]  # [num_valid_objects]
        valid_obj_boxes = pred_boxes[valid_obj_indices]  # [num_valid_objects, 4](x_center, y_center, width, height)

        if not os.path.exists("./val_images"):
            os.mkdir("./val_images")
        if not os.path.exists('./relationship'):
            os.mkdir("./relationship")
        if not os.path.exists(f"./val_images/{batch['id']}.png"):
            image = Image.fromarray(batch['img'], 'RGB')
            filename = f"./val_images/{batch['id']}"
            image.save(f"{filename}.png")
            visualization(image, valid_obj_boxes, filename, valid_obj_classes)
            write_triplets(triplets, batch['id'])

def get_triplets(pred_triplets, classes, boxes):
    triplets, triplet_boxes = [], []
    for sub, obj, rel in pred_triplets:
        triplet_boxes.append([boxes[sub], boxes[obj]])
        sub = id2label[classes[sub]]
        obj = id2label[classes[obj]]
        rel = id2relation[rel]
        triplets.append([obj, rel, sub])
    return triplets, triplet_boxes

def write_triplets(triplets, id):
    file_path = f'./relationship/{id}.txt'
    with open(file_path, 'w') as file:
        for sub, rel, obj in triplets:
            file.write(str(obj) + ' ' + str(rel) + ' ' + str(sub) + '\n')

def get_relation(triplets, id):
    relation_list = []
    for sub, obj, rel in triplets:
        sub = id2label[sub]
        obj = id2label[obj]
        rel = id2relation[rel]
        relation_list.append([sub, obj, rel])
    file_path = f'./relationship/{id}.txt'
    with open(file_path, 'w') as file:
        for sub, obj, rel in relation_list:
            file.write(str(obj) + ' ' + str(rel) + ' ' + str(sub) + '\n')

def visualization(img, box, filename, obj_class):
    colors = ['red', 'blue', 'orange', 'green', 'yellow', 'purple']
    font = ImageFont.truetype("/usr/share/fonts/truetype/tlwg/Garuda.ttf", size=100)
    obj_list = obj_class.cpu().numpy()
    bboxes = box.cpu().numpy()
    draw = ImageDraw.Draw(img)
    font = ImageFont.load_default()
    i = 0
    for bbox, obj in zip(bboxes, obj_list):
        x_center, y_center, width, height = bbox
        x_max = int((x_center + width/2) * img.width)
        y_max = int((y_center + height/2) * img.height)
        x_min = int((x_center - width/2) * img.width)
        y_min = int((y_center - height/2) * img.height)
        obj = id2label[obj]
        text_width, text_height = 10, 10
        text_x = max(x_min + text_width + 5, 0)
        text_y = y_min + text_height + 5
        draw.text((text_x, text_y), str(obj), fill=colors[i], font=font)
        draw.rectangle([(x_min, y_min), (x_max, y_max)], outline=colors[i], fill=None, width=3)
        i = (i + 1)%6
    img.save(f"{filename}_box.png")

Thank you very much for your reply! I need to correspond the output index or label to the actual category, which seems to require id2label and id2elaboration in your code, but I didn't find any relevant information in the code provided by the author. Can you tell me how this was obtained?

@riverjunhao In fact, the author has already provided the id2label and id2relation files. You can take a closer look at this answer, and then modify these two files as needed. I hope this helps. I tested this on the Visual Genome dataset, where the id2label and id2relation variables are loaded when loading the dataset. The code is as follows:
# Dataset
    if "visual_genome" in args.data_path:
        test_dataset = VGDataset(
            data_folder=args.data_path,
            feature_extractor=feature_extractor,
            split=args.split,
            num_object_queries=args.num_queries,
        )
        id2relation = test_dataset.rel_categories
        id2label = {
            k - 1: v["name"] for k, v in test_dataset.coco.cats.items()
        }  # 0 ~ 149
        coco_evaluator = CocoEvaluator(
            test_dataset.coco, ["bbox"]
        )  # initialize evaluator with ground truths
        oi_evaluator = None

@SuperFanBin Thank you very much for your reply! I have tried the code you provided and made some modifications, but there is still an issue: writing information to the specified file requires reading the contents of the batch ['id '] and batch ['img'] keys, which do not seem to be in the 'batch' dictionary. I tried to print the 'batch' dictionary and use the information contained in the [image_id] key under [labels] as the path name to save the file, but I still couldn't extract the content of batch ['img '] well and save it in the image. Could you solve my problem? Thank you very much!


dotnet kernel for C# and F#

This PR adds dotnet kernel to C# and F# languages in jupyter notebooks.
This PR depends on refactorings made in https://github.com/tweag/jupyenv/pull/524
Closes https://github.com/tweag/jupyenv/issues/393

@anpin Thanks for your contributing. once you solve the CI to be green, I will merge your PR.

After merging this PR, we should have a plan to migrate the poetry to the dream2nix. like https://github.com/hardenedlinux/AISecurity-Research-Template/blob/main/flake.nix

@anpin Thanks for your contributing. once you solve the CI to be green, I will merge your PR.

removed dotnet-sdk from requiredRuntimePackages which fixed nix flake check locally

that's weird. can't reproduce these failures locally.

@anpin  Did you run this commandnix build -L .#jupyterlab-kernel-example-dotnet-csharp locally? since I can test them tomorrow night,  please check your test.py, and make sure everything goes well.

yes I do. There is no output in the case of a successful test, so it works fine. I don't have dotnet-sdk in my $PATH

added openssl and zlib to runtimeDeps, maybe that will fix CI

I just realized I have nix-ld on my machine. Maybe that’s why it doesn’t fail. Will try to run without it later.

I disabled nix-ld and reverted the last commit with runtimeDeps, but that did not make it fail. The kernel test executed successfully.
I managed to use lib.mapAttrs' on the example set to run test.py in nixos flake check VM for every kernel. It seems to work for per-existing kernels, but those added in this PR fail with a different error than CI build above.

If running test.py in nix falke check is desired by maintainers I can submit separate a new PR or include here. Here is the gist of it
diff --git a/flake.nix b/flake.nix
index c5d35a5..3a4f4ed 100644
--- a/flake.nix
+++ b/flake.nix
@@ -133,6 +133,40 @@

         exampleJupyterlabAllKernelsNew =
           jupyterLib.mkJupyterlabNew (builtins.attrValues examples);
+
+        checkKernel = name: example:
+          with import (nixpkgs + "/nixos/lib/testing-python.nix") {inherit system;}; let
+            kernel = jupyterLib.mkJupyterlabNew example;
+            test_py = (builtins.dirOf example) + "/test.py";
+          in
+            makeTest {
+              inherit name;
+              nodes.machine = {config, ...}: {
+                config = {
+                  environment.systemPackages = [kernel];
+                };
+              };
+              testScript = ''
+                machine.execute("${kernel}/bin/python ${test_py}")
+              '';
+            };
+        exampleCheck = (
+          lib.mapAttrs'
+          (
+            name: value:
+              lib.nameValuePair
+              ("check-" + name)
+              (checkKernel name value)
+          )
+          examples
+        );
       in {
         lib =
           jupyterLib
@@ -163,9 +197,11 @@
             ${pre-commit.shellHook}
           '';
         };
-        checks = {
-          inherit pre-commit;
-        };
+        checks =
+          {
+            inherit pre-commit;
+          }
+          // exampleCheck;
         apps = {
           update-poetry-lock =
             flake-utils.lib.mkApp

thanks for merging it @GTrunSec!
I've added another commit with above patch which allows to test any kernel via  nix build .\#checks.x86_64-linux.check-example-dotnet-fsharp -L --show-trace. Would you consider such change in a new PR?


Scenario switcher - htm

This PR builds from PR #1273 and adds a set of scale-up scenarios hosted within the HIV module. These scenarios can switch key HIV, TB or malaria parameters to meet targets at a specified date.
The new scale-up parameters are defined in ResourceFile_HIV, sheet scaleup_parameters
HIV parameters meet the 95-95-95 targets
TB parameters meet the 90s targets
Malaria parameters follow the Malawi National Malaria Communication Strategy 2022-2030. Note the limit for ITN coverage is 0.7 (set by the lookup tables which have 0.7 as max coverage) and IRS can be either 0 or 0.8. Therefore neither can be set to 100% coverage. ITN household coverage could be assumed to be 100%, but only 70% of householders sleep under the bed-net.
test_HTMscaleup.py checks the parameters are set correctly and scaled-up appropriately. These tests are not exhaustive - not every parameter is checked and not every combination of scenarios are checked. A selection of parameters are checked for each scenario.
scripts/malaria/analysis_malaria.py can be used to run the scenarios
TO DO: once PR #1273 finalised and in master, pull these changes in to avoid merge conflicts.

Now that #1273 is in, next job on this is to merge in master (currently there are conflicts). Then I can review and bring it in.

I'll be able to look at this properly when we have master merged in any other incidental changes rolled back but from a quick look (and if I'm not getting confused) some first reactions are:

The switches seem to be in the form of key-word arguments in the Hiv module. However, the Scenario base class that we use to develop the scenarios work by switching Parameters and can't touch the keyword arguments to a module. So, I think we need to bring those switches into being Parameters.

Is there a very strong design argument for having the switches for HTM all in the Hiv module, or was this just the most convenient for other reasons? It might be ok, and it's not worth doing some massive refactoring, if that's what it take, but, I suppose that in an ideal world, there would be a switch in each module that controls it's own scale-up (to decouple the module as much as possible). e.g. each of the HTM modules has a Parameter that is of the form:

'do_scale-up':  Parameter(Types.Bool, "Whether services for this disease should scale-up following the 'global plans'.")

It would be really useful if you could create a single file with a Scenario defined within it that show me how to use these switches. e.g. A Scenario that runs a 'Status Quo' projection and a scenario that has the HTM scale-up. I'll then be able to incorporate it into other scripts we have.

And, lastly, it would great if you could send that off the Batch and get it run at a decent size (100k pop, 5 repeats, 2010-2030), and paste into this PR how the impact looks (wrt Deaths/DALYs due to HTM/other cumulatively in the period 2020-2030).


Weird situation with a table column's value fetching

Laravel Version: 5.4
PHP Version: PHP 7.1.0
Database Driver & Version: MariaDB

I'm retrieving data from Database like everyone else, but I'm facing a weird issue.
I'm using the slug in my table to retrieve the element's data but but I display the slug it gives me nulland using the famous dd()famous it shows up here is an example :
dd($element);

##Result

dd($snippets->toArray());

##Result

##Table
Schema::create('elements', function (Blueprint $table) {
    $table->increments('id');
    $table->integer('user_id');
    $table->string('title');
    $table->string('slug')->unique();
    $table->text('body');
    $table->timestamps();
});

##Eloquent
$snippets = Snippet::latest()->with('owner')->get();

##Snippet Model
protected $primaryKey = 'slug';

public function owner()
{
    return $this->belongsTo(User::class, 'user_id');
}

Is there any one who knows what is going on?

You set 'slug' as primary key and have not set incrementing to false which is causing it to 'cast' it by the keyType.  The keyType is set to 'int' unless set otherwise.

You set 'slug' as primary key and have not set incrementing to false which is causing it to 'cast' it by the keyType.  The keyType is set to 'int' unless set otherwise.

Yes, I've figured it out and I forgot to close the issue.
Thank you so much

Yes, I've figured it out and I forgot to close the issue.
Thank you so much


Programatically trigger animation

Hello,
I have been using SWTableViewCell in a project but I wanted to make the user aware that the swipe function is available on a particular UITableview. I am hoping to do this by momentarily revealing the swipe view animation of the first tableviewcell when the tableview is loaded.  So for example the tableview will load, the right utility Button will open for cell 0, then momentarily hide again - thereby showing the user that there is an option to swipe the cell to reveal a 'right utility button'.  I am thinking that could achieve this by using a simple timer and by triggering the appropriate methods.
However, I need some direction as to which methods to call for this to work.  I have tried the following but this has not actioned the animation itself:

NSIndexPath* indexPath= [NSIndexPath indexPathForRow:0 inSection:0];
GenericTableViewCell6 *cell = (GenericTableViewCell6 *)[self.tableView cellForRowAtIndexPath:indexPath];
[ self swipeableTableViewCell:cell scrollingToState:2];

I have tried tweaking this code to see if i can action the animation - but again this tweak has not worked for me.

NSIndexPath* indexPath= [NSIndexPath indexPathForRow:0 inSection:0];
GenericTableViewCell6 *cell = (GenericTableViewCell6 *)[self.tableView cellForRowAtIndexPath:indexPath];
[cell showRightUtilityButtonsAnimated:YES];

I would be very grateful for any guidance or input on this.  please message if you need any further information.
many thanks
james.

I have solved this problem. Here is my code for others that wish to achieve a similar effect:
- (void)tableView:(UITableView *)tableView willDisplayCell:(SWTableViewCell *)cell forRowAtIndexPath:(NSIndexPath *)indexPath
{
    if (indexPath.row== 0) {
        [cell showRightUtilityButtonsAnimated:YES];

        double delayInSeconds = 2.0;
        dispatch_time_t popTime = dispatch_time(DISPATCH_TIME_NOW, delayInSeconds * NSEC_PER_SEC);
        dispatch_after(popTime, dispatch_get_main_queue(), ^(void){
        [cell showLeftUtilityButtonsAnimated:YES];

        });

    }

}


Clicking on next randomly jumps backwards or changes direction to previous

This is a (multiple allowed):

[x] bug

[ ] enhancement

[ ] feature-discussion (RFC)

Swiper Version: 4.5.0

Platform/Target and Browser Versions: Windows 10, Chrome 74.0.3729.157

Live Link or JSFiddle/Codepen or website with isssue: http://idangero.us/swiper/demos/020-navigation.html

What you did
Multiple fast clicking on next slide.
Expected Behavior
Change to next slide.
Actual Behavior
Randomly jumping back to a few slides or changing direction to backwards (so next acts like previous).
It seems quite random, but happens with a 20-30% chance, I have tried many combinations of options but it didn't help, also it acts similar with the simple demo too.

Hi, I'm having the same issue, my case is special, I'm on Wordpress, working on a custom theme, but when working with the lazy load WP Rocket plugin, it's bugged, but when the plugin is desactivated, all good =)

also experiencing this randomly at times

I am also experiencing this. Has anyone found a fix?

Because you also do a short swipes, use touchEventsTarget: 'wrapper'


feat: Upgraded admin token to unlimited duration

This PR removes the expiration of admin tokens so that master services (such as the platform) don't get issues with API access.
Any feedback is welcome!

This is actually quite dangerous, closing this in favour of #180


Gracefully handle malformed UUIDs in _id

In most places where querying by ID, we properly validate the UUID input string.  I.e., read by ID, update by ID, etc.
When using the special _id search parameter, we did not correctly validate the UUID string.  For a well formed UUID, everything was fine.  For an invalid UUID, postgres would throw an error.  So rather than empty result, the server threw a 500.
This fixes that by treating malformed UUID's as the special "all zero UUID", which should never match.

Coverage: 94.007% (+0.002%) from 94.005% when pulling e4da6b10776cca238b30609aac763ad3ea3af515 on cody-handle-malformed-uuids into ad7554aaf095d946e33a55cb170118969af24fb4 on main.


Portus LDAP doesnt work

I am using portus and i have configured LDAP to authenticate users and i have normal users as well in Portus. Eventhough i have LDAP enabled in my config-local.yml file the login page still doesnt comes to LDAP LOGIN button.
Version of Portus : Portus-20151120162040-2.2.x86_64
LDAP SERVER : Active Directory
Kindly help with this issue. Reference of my config-local.yml
ldap:
enabled: true
hostname: emea.corpdir.net
port: 389
method: "plain"
base: "dc=my,dc=ad,dc=server"
uid: "cn"
guess_mail:
enabled: true
attr: ""
authentication:
enabled: true
bind_dn: "cn=userid,ou=admins,ou=resources,ou=docker,dc=my,dc=ad,dc=server"
password: "blahblah"
first_user_admin:
enabled: true
check_ssl_usage:
enabled: true
signup:
enabled: true
machine_fqdn:
value: "serverhostname.with.fqdn"
Note: When i use my ldap bind dn username and password it works perfectly fine. I am able to fetch users. No issue in my configuration.
Can someone help me with this or route me to a right channel where i can get help.

Hi @arjunm183, note that we do not allow to mix regular users with LDAP users. Doing so is undefined behavior :)

Note: When i use my ldap bind dn username and password it works perfectly fine. I am able to fetch users. No issue in my configuration.

Are you saying that if you try to authenticate through LDAP in Portus (regardless of what the button says) it works ? That is weird... Are you sure that Portus is picking the right configuration ? As in, did you restart Portus when you introduced your config-local.yml file ?

Thanks for the Reply Portus geeks.
So if i understood you correct, you mean either we should use LDAP users or normal users for Portus Login.
No it doesn't work when i try to login with LDAP user in Portus, even if button is not showing as LDAP login, however normal users login work.
After ldap configuration in Portus, i have restarted the service named portus_crono.
systemctl portus_crono restart
But still the ldap login page didnt came. I have posted the configuration of config-local.yml in my previous post as well. Please help me in fixing this.

So if i understood you correct, you mean either we should use LDAP users or normal users for Portus Login.

That is correct. If you have LDAP enabled, then only users authenticated through the LDAP server are allowed to enter. If you previously set up Portus without LDAP and entered some users, these users are probably not usable any more. I say probably because that's what I expect, but it's undefined behavior.

After ldap configuration in Portus, i have restarted the service named portus_crono.
systemctl portus_crono restart

This is just the crono service. You should restart Portus itself. You can do that by restarting Apache, NGinx or whatever method you are using to deploy Portus.

I have deployed in apache2 so now hurray i am able to see the LDAP Login page.
When i try to login with my ldap user id it doesnt login.
However when i try to fetch the ldapuser with the given attr in the portus with ldapsearch command
it shows the details of the user.
Can you suggest.

Maybe you can check what are the logs saying ? Maybe there's some value wrong on your config-local.yml file ? Maybe it's because of Active Directory (we are not doing anything specific for AD) ?

I recommend you to use sAMAccountName as uid and mail for the attr in guess_mail.
Here is a working example for authenticating against active directory:
ldap:
  enabled: true

  hostname: "mydomain.com"
  port: 389

  method: "plain"

  base: "dc=mydomain,dc=com"

  uid: "sAMAccountName"

  authentication:
    enabled: true
    bind_dn: "cn=service-domain-user-for-portus,cn=Users,dc=mydomain,dc=com"
    password: "the-correct-password"

  guess_email:
    enabled: true
    attr: "mail"

Here is my config-local.yml file
ldap:
enabled: true
hostname: AD Server name
port: 389
method: "plain"
base: "dc=fully,dc=qualified,dc=domain"
uid: sAMAccountName
authentication:
enabled: true
bind_dn: "cn=username,ou=portus,ou=ouname,ou=ext,dc=fully,dc=qualified,dc=domain"
password: "blahblah"
guess_mail:
enabled: true
attr: "mail"
first_user_admin:
enabled: true
By default require ssl to be enabled when running on production
check_ssl_usage:
enabled: true
When i enter the username it says INvalid username or password.
But as i said in my previous quotes ldapsearch command i am able to query the username with sAMAccountName
Additional Information: I have config.yml file as well in the config folder for portus. Does it matters if i have two yml config files " config.yml and config-local.yml "

I have config.yml file as well in the config folder for portus. Does it matters if i have two yml config files " config.yml and config-local.yml

This doesn't matter. The contents of config-local.yml override config.yml. See the documentation page for this.

My ldap user name contains special character

That doesn't matter either. Portus just acts as a proxy when it comes to LDAP authentication.

But as i said in my previous quotes ldapsearch command i am able to query the username with sAMAccountName

When you do that, do you also use the authentication bind_dn that you provide to Portus ? Just checking whether what you have wrong is the bind_dn or the password for that (note that in some servers you can event have that disabled).

Yes, ofcourse when i use ldapsearch command i use the same bind_dn and password which i have provided to Portus as well.

I wanted to know whether i have did something wrong
Initially my portus was running in normal user id and then i tried configuring LDAP ... Is there something which i have to do in such kind of cases.

@arjunm183 you really want to start fresh with your LDAP configuration (that is, without entering a user that doesn't go through LDAP). If you don't want to start all over again, maybe you can try running on debug mode as explained in the last point of this section.
Otherwise, maybe you can try to run another Portus instance with a fresh DB and configured with LDAP from the beginning (but I'd recommend running on debug mode before doing anything crazy :P)

I tried in debug mode. However couldn't get much information in the log file (other than DB query commands) mentioned with debug tag
Planning to run another portus instance with fresh DB configuration.
I doubt if we initially setting up portus either we have to configure with normal users or LDAP users.
Changing from Normal to LDAP doesnt work. Let me give a try and update you ..

@mssola Hi i tried setting up seperate portus instance and from the beggining i am using ldap only
here is the configuration of ldap
ldap:
enabled: false
ldap:
enabled: true
hostname: "ema.corp.net"
port: 389
method: "starttls"
base: "dc=ema,dc=corp,dc=net"
uid: "sAMAccountName"
authentication:
enabled: true
bind_dn: "cn=E117_s_ldap-lookup,ou=UsersAdministrative,ou=_GlobalResources,ou=e1,dc=ema,dc=corp,dc=net"
password: "blahblah"
guess_mail:
enabled: true
attr: "mail"
It shows ldap login and when i entered by userid it says invalid user name and password
my user name is similar to this
username : ecd_a_arjun
I am able to fetch my user id using ldapsearch command in linux
Kindly help on this.

More information on this issue . Production logs for portus as follows
I, [2016-05-16T07:54:46.775397 #17832]  INFO -- : Completed 401 Unauthorized in 251ms (ActiveRecord: 0.0ms)
I, [2016-05-16T07:54:46.776802 #17832]  INFO -- : Processing by Auth::SessionsController#new as HTML
I, [2016-05-16T07:54:46.776974 #17832]  INFO -- :   Parameters: {"utf8"=>"✓", "authenticity_token"=>"ksjC1m+tXu/+7nEkpHtPnyS67Z1h5u9RWdAcumpwQWWXnKbV5g19p0J/45NhMa8rn5362kyZbcJYJ65MQ9CY6A==", "user"=>{"username"=>"ECD_a_ARJUN", "password"=>"[FILTERED]"}, "button"=>""}
I, [2016-05-16T07:54:46.915777 #17832]  INFO -- :   Rendered shared/_notification.html.slim (0.1ms)
I, [2016-05-16T07:54:46.916229 #17832]  INFO -- :   Rendered shared/_notification.html.slim (0.1ms)
I, [2016-05-16T07:54:46.916539 #17832]  INFO -- :   Rendered shared/_notifications.html.slim (1.1ms)
I, [2016-05-16T07:54:46.920306 #17832]  INFO -- :   Rendered devise/sessions/new.html.slim within layouts/authentication (5.1ms)
I, [2016-05-16T07:54:46.921610 #17832]  INFO -- : Completed 200 OK in 144ms (Views: 7.0ms | ActiveRecord: 0.7ms)

@arjunm183 Did you find a fix for the ldap issue? Can you post your finding if possible.

@mssola @here I turned on debug mode to troubleshoot LDAP issue. Here is the error message
Processing by Auth::SessionsController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"Z7YLm1wQqB9M4B06rbxB29ccbYqMbfH6XvZJCgs3bvwonURohCwD2z+MT/TqNppmfZ6n43QisXcuqFHTkOQHvw==", "user"=>{"username"=>"gvenka008c", "password"=>"[FILTERED]"}, "button"=>""}
Completed 401 Unauthorized in 22ms (ActiveRecord: 0.0ms)
Processing by Auth::SessionsController#new as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"Z7YLm1wQqB9M4B06rbxB29ccbYqMbfH6XvZJCgs3bvwonURohCwD2z+MT/TqNppmfZ6n43QisXcuqFHTkOQHvw==", "user"=>{"username"=>"gvenka008c", "password"=>"[FILTERED]"}, "button"=>""}
   (0.5ms)  SELECT COUNT(*) FROM `users` WHERE (`users`.`username` != 'portus')
  Rendered shared/_notification.html.slim (0.4ms)
  Rendered shared/_notification.html.slim (0.2ms)
  Rendered shared/_notifications.html.slim (2.0ms)
  CACHE (0.0ms)  SELECT COUNT(*) FROM `users` WHERE (`users`.`username` != 'portus')  [["username", "portus"]]
  Rendered devise/sessions/new.html.slim within layouts/authentication (7.2ms)
Completed 200 OK in 126ms (Views: 35.7ms | ActiveRecord: 0.5ms)

Any way to find out if the bind was successful? Please suggest. Thanks.

We don't really log error messages on unsuccessful binds. So, my guess is that either your credentials are not correct, or the LDAP server is not reachable somehow, etc. Either way, maybe you could investigate this in the logs of your LDAP server if you have access to it ?

A lot has happened meanwhile and I believe this issue is no longer relevant in master. If it is, please re-open this issue, or we can proceed this discussion in our mailing list. Thanks for the feedback 👍


puppet: Configure nginx for long keep-alives when behind a loadbalancer.

These optimizations only makes sense when all connections at a TCP
level are coming from the same host or set of hosts; as such, they
are only enabled if loadbalancer.ips is set in the zulip.conf.
Testing plan: Test-deployed to staging.

Ref https://aws.amazon.com/premiumsupport/knowledge-center/apache-backend-elb/

I assume there will be a future commit to configure using these.

Not sure what you mean here -- by "configure using these" do you mean "make these values configurable"?  If you mean "make nginx include these," that's not a necessary extra step; by being in /etc/nginx/zulip-include/app.d the file is included by nginx just by being put in place.

Ahh, right, I was thinking the parent directory.  sgtm.  I guess it's good that these values are consistent with what we recommend in docs/production/deployment.md, since this configuration should be correct for anyone self-hosting as well.


It's not possible to connect to Azure Data Lake Storage diagnostic logs.

Azure Data Lake Storage Gen2 can be configured to capture diagnostic logs. The logs are written to the system controlled $logs container.
The PBI - Azure Data Lake Storage Gen2 connector (in preview) uses a URL end-point to connect to ADLS Gen2. To read the $logs container the URL would need to be:  https://myresourcename.blob.core.windows.net/$logs
However, the connector considers the URL to be invalid.
Details: "The Container name must be 3 to 63 lower case letters and numbers. It must start with a letter or number and can contain only letters, numbers, and the dash (-) character. No consecutive dashes are allowed. Every dash (-) character must be immediately preceded and followed by a letter or number."
Thanks,
Gary

Thanks for your comment, @GaryStrange . This looks like a support request or product question rather than documentation feedback. You might find a solution in the Power BI Community, at https://community.powerbi.com. If not, you can get support by going to https://support.powerbi.com.


Source configuration from pyproject.toml

Description
Current usage is djade --target-version 5.1 templates/**/*.html. It would be awesome if djade could be configured in pyproject.toml like so:
[tool.djade]
files = [
    "templates/**/*.html"
]
target_version = "5.1"

I won't add configuration for which files to run on. I don't believe it's productive for every tool to rebuild their own custom globbing and recursion logic.
But yes, configuring the target version would be neat. There's an idea in django-upgrade to support parsing the target version from dependencies in pyproject.toml ( https://github.com/adamchainz/django-upgrade/issues/322 ). If we can figure that out, it would be worth adding here.


Add the JSON Schema test suite

We should be able to parse all valid test cases.
https://github.com/json-schema-org/JSON-Schema-Test-Suite

hi, @schani can I give it a try?

Hi @snehaljos!  For sure!
There are two main parts to this:

You'll have to write a driver (I suggest adding an extra command line option to main in test.ts) that loops through all the JSON Schema test files, and for each test schema in each of the files, run quicktype and produce JS output.  It shouldn't crash.
Now for each of the schemas, run the JS output on each of the tests that has valid: true.  They should all succeed.


Plugin Respond Issue

2024-07-17.txt
Flow Launcher version: 1.18.0
OS Version: 19045.4651
IntPtr Length: 8
x64: True
Python Path: C:\Users\admin\AppData\Roaming\FlowLauncher\Environments\Python\PythonEmbeddable-v3.11.4\pythonw.exe
Node Path:
Date: 07/17/2024 10:39:03
Exception:
ElementFlow Exception:
Websites: https://github.com/asmpro7/ElementFlow
Author: Ahmed ElSaeed
Version: 1.0.4
Flow.Launcher.Core.ExternalPlugins.FlowPluginException: Traceback (most recent call last):
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy\core_init_.py", line 23, in
from . import multiarray
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy\core\multiarray.py", line 10, in
from . import overrides
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy\core\overrides.py", line 6, in
from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\main.py", line 13, in
import numpy as np
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy_init_.py", line 141, in
from . import core
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy\core_init_.py", line 49, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

The Python version is: Python3.11 from "C:\Users\admin\AppData\Roaming\FlowLauncher\Environments\Python\PythonEmbeddable-v3.11.4\pythonw.exe"
The NumPy version is: "1.24.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy.core._multiarray_umath'
---> System.IO.InvalidDataException: Traceback (most recent call last):
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy\core_init_.py", line 23, in
from . import multiarray
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy\core\multiarray.py", line 10, in
from . import overrides
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy\core\overrides.py", line 6, in
from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\main.py", line 13, in
import numpy as np
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy_init_.py", line 141, in
from . import core
File "C:\Users\admin\AppData\Roaming\FlowLauncher\Plugins\ElementFlow-1.0.4\lib\numpy\core_init_.py", line 49, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

The Python version is: Python3.11 from "C:\Users\admin\AppData\Roaming\FlowLauncher\Environments\Python\PythonEmbeddable-v3.11.4\pythonw.exe"
The NumPy version is: "1.24.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy.core._multiarray_umath'
at Flow.Launcher.Core.Plugin.JsonRPCPlugin.ExecuteAsync(ProcessStartInfo startInfo, CancellationToken token) in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\JsonRPCPlugin.cs:line 243
at Flow.Launcher.Core.Plugin.JsonRPCPlugin.ExecuteAsync(ProcessStartInfo startInfo, CancellationToken token) in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\JsonRPCPlugin.cs:line 248
at Flow.Launcher.Core.Plugin.JsonRPCPlugin.QueryAsync(Query query, CancellationToken token) in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\JsonRPCPlugin.cs:line 261
at Flow.Launcher.Core.Plugin.PluginManager.<>c__DisplayClass23_0.<b__0>d.MoveNext() in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\PluginManager.cs:line 196
--- End of stack trace from previous location ---
at Flow.Launcher.Infrastructure.Stopwatch.DebugAsync(String message, Func`1 action) in C:\projects\flow-launcher\Flow.Launcher.Infrastructure\Stopwatch.cs:line 34
at Flow.Launcher.Core.Plugin.PluginManager.QueryForPluginAsync(PluginPair pair, Query query, CancellationToken token) in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\PluginManager.cs:line 195
--- End of inner exception stack trace ---
at Flow.Launcher.Core.Plugin.PluginManager.<>c__DisplayClass23_1.b__1(ActionContext ) in C:\projects\flow-launcher\Flow.Launcher.Core\Plugin\PluginManager.cs:line 224
at Flow.Launcher.Plugin.Result.ExecuteAsync(ActionContext context) in //Flow.Launcher.Plugin/Result.cs:line 246
at Flow.Launcher.ViewModel.MainViewModel.OpenResultAsync(String index) in C:\projects\flow-launcher\Flow.Launcher\ViewModel\MainViewModel.cs:line 307
at CommunityToolkit.Mvvm.Input.AsyncRelayCommand.AwaitAndThrowIfFailed(Task executionTask)
at System.Threading.Tasks.Task.<>c.b__128_0(Object state)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

Your error is stated here:
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
In the folder python 3.11, a "numpy library" needs to be installed.
I suggest you install a Python version on your own (not auto installation by Flow Launch), then point the Python exe in the General tab setting in Flow Launcher.
Hope it help.

Hello I also made a plugin for looking up elements in the periodic table. It shows way less info but it also uses the same .json file or at least a file from the same repository. It worked pretty well but @Jalaj1087 requested to implement search by atomic number. I don't know why but I looked at his profile (normally I don't do this because it feels like I'm stalking someone but for some reason I checked it out) and found this issue, I was like huh another periodic table plugin, but I searched for this and never found one. It's because I was searching for periodic.... but never chemical....
So I found this plugin and tested it, I already knew it wouldn't work but tried anyway. The problem was numpy, an explanation is found in my pull request #2. After a search in the code I found numpy was only used twice and both were to create the image of the element color. When removing all code for the images, I found this plugin was way better than mine so I wanted to try and make it work with the build in python of flowlauncher.
Then I tried searching for solutions for the numpy problem. And guess what, the solution was completely no numpy, it is a library purely written in python: https://gitlab.com/drj11/pypng/-/tree/main?ref_type=heads. So I implemented a modified version of the image function and now it should work, at least according to my testing.
TLDR: This problem has a fix. And in my opinion a better one than installing python yourself.


The proxy shouldn't ask the controller to resolve IP addresses, especially localhost addresses <IP_ADDRESS> and ::1

I understand that at some point in the future we might try to implement some interesting logic when the proxy gets an IP address as the destination, and that interesting logic would probably be better implemented in the controller. However, I think the implementation of that logic is quite far out and maybe won't ever be needed. Unless/until we get closer to that, we should just have the proxy resolve IP addresses internally, without contacting the controller.
In particular, we definitely shouldn't ask the controller to resolve <IP_ADDRESS> and ::1. Applications use these loopback addresses to guarantee that their networking traffic is local for security reasons (especially) and performance reasons. The interesting logic we intend to support by passing IP addresses to the controller won't be able to be done for loopback addresses without serious risk of violating the security assumptions the proxied application is making. Avoiding resolving these addresses through the controller is an important step towards ensuring the proxy doesn't violate any security assumptions.
/cc @seanmonstar @adleong

:+1:

The changes in #397 means that the destination service is no longer asked about IP addresses in the authority. However, it doesn't do anything with special addresses.

I think we can call this "Fixed" by just adding some more none() tests that cover IP addresses (IPv4 and IPv6), especially but not only <IP_ADDRESS> and ::1, to test_normalized_authority().
(localhost and other non-IP-address issues are described elsewhere.)


Update swift-cmark reference to swiftlang org

We're seeing issues with GitHub failing to follow the redirect, so it might help to get this updated to the new org.

@swift-ci Please test


Config changes to reflect the default branch from master to main. Please merge to complete the default branch renaming job.

Auto branch renaming content.

Docs Build status updates of commit 58a6721:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch 406a1d3e-9848-4369-be4d-f0f76b1deb37 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel