Can't archive iOS app when Swift plugins are used

I'm working on an app which uses the geolocation 0.2.1 Swift plugin and the cloud_firestore 0.7.3. The app compiles without an error when deploying to the simulator or an connected iOS device. However, when I try to 'archive' the app in order to upload it to the AppStore, I get the following error message 'cloud_firestore/CloudFirestorePlugin.h' file not found.
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel dev, v0.5.4, on Mac OS X 10.13.5 17F77, locale en-DE)
[✓] Android toolchain - develop for Android devices (Android SDK 28.0.0-rc1)
[✓] iOS toolchain - develop for iOS devices (Xcode 9.4.1)
[✓] Android Studio (version 3.1)
[✓] IntelliJ IDEA Community Edition (version 2018.1.3)
[!] VS Code (version 1.24.1)
[✓] Connected devices (1 available)

! Doctor found issues in 1 category.

I'm using Cocoapods 1.5.3.
Podfile:
# Uncomment this line to define a global platform for your project
platform :ios, '9.0'

# CocoaPods analytics sends network stats synchronously affecting flutter build latency.
ENV['COCOAPODS_DISABLE_STATS'] = 'true'

def parse_KV_file(file, separator='=')
  file_abs_path = File.expand_path(file)
  if !File.exists? file_abs_path
    return [];
  end
  pods_ary = []
  skip_line_start_symbols = ["#", "/"]
  File.foreach(file_abs_path) { |line|
      next if skip_line_start_symbols.any? { |symbol| line =~ /^\s*#{symbol}/ }
      plugin = line.split(pattern=separator)
      if plugin.length == 2
        podname = plugin[0].strip()
        path = plugin[1].strip()
        podpath = File.expand_path("#{path}", file_abs_path)
        pods_ary.push({:name => podname, :path => podpath});
      else
        puts "Invalid plugin specification: #{line}"
      end
  }
  return pods_ary
end

target 'Runner' do
  use_frameworks!

  # Prepare symlinks folder. We use symlinks to avoid having Podfile.lock
  # referring to absolute paths on developers' machines.
  system('rm -rf .symlinks')
  system('mkdir -p .symlinks/plugins')

  # Flutter Pods
  generated_xcode_build_settings = parse_KV_file('./Flutter/Generated.xcconfig')
  if generated_xcode_build_settings.empty?
    puts "Generated.xcconfig must exist. If you're running pod install manually, make sure flutter packages get is executed first."
  end
  generated_xcode_build_settings.map { |p|
    if p[:name] == 'FLUTTER_FRAMEWORK_DIR'
      symlink = File.join('.symlinks', 'flutter')
      File.symlink(File.dirname(p[:path]), symlink)
      pod 'Flutter', :path => File.join(symlink, File.basename(p[:path]))
    end
  }

  # Plugin Pods
  plugin_pods = parse_KV_file('../.flutter-plugins')
  plugin_pods.map { |p|
    symlink = File.join('.symlinks', 'plugins', p[:name])
    File.symlink(p[:path], symlink)
    pod p[:name], :path => File.join(symlink, 'ios')
  }
end

post_install do |installer|
  installer.pods_project.targets.each do |target|
    target.build_configurations.each do |config|
      config.build_settings['ENABLE_BITCODE'] = 'NO'
    end
  end
end

Also tried with the current master but still the same problem:
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel master, v0.5.6-pre.27, on Mac OS X 10.13.5 17F77, locale en-DE)
[✓] Android toolchain - develop for Android devices (Android SDK 28.0.0-rc1)
[✓] iOS toolchain - develop for iOS devices (Xcode 9.4.1)
[✓] Android Studio (version 3.1)
[✓] IntelliJ IDEA Community Edition (version 2018.1.3)
[✓] VS Code (version 1.24.1)
[✓] Connected devices (1 available)

• No issues found!

Did no one else face this problem?

@Hixie

I canreproduce it. Here is what I did

Create project:

> flutter create flutter_geo_release_bug
> cd flutter_geo_release_bug

add geolocation: 0.2.1 and cloud_firestore 0.7.3 dependency followed by flutter pacakges get
Adjust signing in Xcode
Set SWIFT_VERSION to 3.2
Create release flutter build ios --release

Error:
Building net.grandcentrix.flutterGeoBug for device (ios-release)...
Automatically signing iOS for device deployment using specified development team in Xcode project: JCPY7FXGR4
Running pod install...
Xcode build done.
Failed to build iOS app
Error output from Xcode build:
↳
    ** BUILD FAILED **

Xcode's output:
↳
    /Users/pascalwelsch/.pub-cache/hosted/pub.dartlang.org/geolocation-0.2.1/ios/Classes/GeolocationPlugin.m:7:9: fatal error: 'geolocation/geolocation-Swift.h' file not found
    #import <geolocation/geolocation-Swift.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
Encountered error while building for device.

Found the root cause: By default flutter generates an Objective-C iOS project. Integration Swift dependencies isn't straight forward. By generating a Swift project, the procedure above works flawless.

Create project:

> flutter create -i swift flutter_geo_release_bug
> cd flutter_geo_release_bug

add geolocation: 0.2.1 and cloud_firestore: 0.7.3 dependency followed by flutter pacakges get
Adjust signing in Xcode
Create release flutter build ios --release
Select "Generic iOS Device"
Product > Archive > Success 🎉

I don't know if it's possible to convert an Objective-C project to a Swift project but you'll find a way @flschweiger

https://github.com/flutter/flutter/issues/13422#issuecomment-392133780

Thanks for all your effort @passsy 😃 Unfortunately that's not the issue. I already have a Swift initialised project with a valid AppDelegate.swift file. Also created a clean Flutter project and copied everything over - still the same issue 😔

cc @mravn-google, @collinjackson

@flschweiger Does your project contain an ios/Podfile which is different from the one brought by Flutter tooling?

@mravn-google I have set the global platform to 9.0 by uncommenting the second line in the default podfile (which is a requirement for the geolocation library). Otherwise I didn't change anything.
If needed I can also provide access to the repo.

Any update on this? This also fails when I create a new objective c project which uses Swift plugins (in this case, the geolocation plugin which requires iOS 9 and the Firestore).

Finally found the issue, it was my fault 🙈 I had opened the Runner.xcodeproj instead of the Runner.xcworkspace file - as an Android developer I didn't know that there was a difference between those two when opened in Xcode.

Finally found the issue, it was my fault 🙈 I had opened the Runner.xcodeproj instead of the Runner.xcworkspace file - as an Android developer I didn't know that there was a difference between those two when opened in Xcode.

After implementing the steps mentioned by @passy, the first 3 app builds went on without a hassle, then after that, back to this.
/Users/rosius/flutter/.pub-cache/hosted/pub.dartlang.org/geolocation-0.2.1/ios/Classes/GeolocationPlugin.m:7:9: fatal error: 'geolocation/geolocation-Swift.h' file not found
#import <geolocation/geolocation-Swift.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The files for my project are too many.
Please i desperately need your help on this.

@trey-rosius Did you open the workspace file instead of the project file?


Fix a possible memory leak of ssl->s3.tmp.psk

It looks like the git cherry-pick is working for 3.1 & 3.0.
Only 1.1.1 would need a different PR due to the s3 being a pointer.

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Merged to all branches.  Thanks!

Fix in #22682


Can I use a single gpu to train this model?

Before you open an issue, please check if a similar issue already exists or has been closed before.
When you open an issue, please be sure to include the following

[x] A descriptive title:  Can I use a single gpu to train this model?
[x] A detailed description: Thanks for your nice work. I want to train the Otter model. May I use a single GPU to train the model. Could you please share your accelerate config? Thanks!
[ ] Assign an issue type tag (label):

dataset (mimic-it download, usage, etc.),
demo (online demo), doc (readme, wiki, paper, video etc.),
evaluation (evaluation result, performance of Otter etc.),
model (model configuration, components, etc.),
train (training configuration, process, code, etc.)

Thank you for your contributions!

We put our configs here.
You can use 1 GPU with at least 33-34G memory to start training. If you load the model with bf16 or fp16, you can lower the GPU requirement to 16-20G I think. But we didnt try to train a model under bf16 or fp16 (we actually use mixed precision).
You can refer the following code to load a model under certain precisions.
import os
import torch
import argparse
from .configuration_flamingo import FlamingoConfig
from .modeling_flamingo import FlamingoForConditionalGeneration

parser = argparse.ArgumentParser(description="Load model with precision")
parser.add_argument('--load_bit', type=str, choices=['fp16', 'bf16'], required=True, help="Choose either 'fp16' or 'bf16'")
args = parser.parse_args()

load_bit = args.load_bit

if load_bit == "fp16":
    precision = {"torch_dtype": torch.float16}
elif load_bit == "bf16":
    precision = {"torch_dtype": torch.bfloat16}

root_dir = os.environ["AZP"]
print(root_dir)
checkpoint_path = f"{root_dir}/otter/checkpoints/flamingo-mpt-7B-instruct-init"
device_id = "cpu"
model = FlamingoForConditionalGeneration.from_pretrained(checkpoint_path, device_map={"": device_id}, **precision)

# save model
checkpoint_path = checkpoint_path + f"_{load_bit}"
model.save_pretrained(checkpoint_path)

Got it. Thanks a lot!

Hi Bo, I tried the following script, but I got an error. I want to train the model using LA dataset and I added the image_processor and train_num_samples back to the args. The command I used is
accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml
 pipeline/train/instruction_following.py \
--pretrained_model_name_or_path=/path/to/OTTER-9B-INIT \
--mimicit_path="/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_I2I_instructions.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_T2T_instructions.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACONV_instructions.json,/path/toOtter/OneDrive_1_6-27-2023/LA/LADD_instructions.json" \
--images_path="/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json" \
--train_config_path="/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_I2I_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_T2T_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACONV_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LADD_train.json" \
--batch_size=16 \
--num_epoch=6 \
--external_save_dir=./checkpoints \
--save_hf_model \
--workers=8 \
--cross_attn_every_n_layers=4 \
--lr_schedule=cosine \
--learning_rate=1e-5 \
--warmup_steps_ratio=0.01

The error I got is
Traceback (most recent call last):
  File "/path/to/Otter/pipeline/train/instruction_following.py", line 498, in <module>
    main()
  File "/path/to/Otter/pipeline/train/instruction_following.py", line 451, in main
    model, optimizer, lr_scheduler, mimicit_loaders = accelerator.prepare(model, optimizer, lr_scheduler, mimicit_loaders)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1182, in prepare
    result = tuple(
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1183, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1022, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1311, in prepare_model
    torch.autocast(device_type=self.device.type, dtype=torch.bfloat16)(model.forward.__func__), model
AttributeError: 'function' object has no attribute '__func__'

Could you please give me some suggestions? I am very new to the accelerate package.
The GPU I used is A100 and I think it is not the GPU isssue.
Thanks!

Hi May I know if you properly install the packages, how about the accelerate, transformers and torch version?

Yes. My software versions are as follows.
transformers: 4.30.2
accelerate: 0.20.3
torch: 2.0.1
Sure. I can try the latest branch.
Thanks!

Hi Bo, I tried the following script, but I got an error. I want to train the model using LA dataset and I added the image_processor and train_num_samples back to the args. The command I used is
accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml
 pipeline/train/instruction_following.py \
--pretrained_model_name_or_path=/path/to/OTTER-9B-INIT \
--mimicit_path="/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_I2I_instructions.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_T2T_instructions.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACONV_instructions.json,/path/toOtter/OneDrive_1_6-27-2023/LA/LADD_instructions.json" \
--images_path="/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json" \
--train_config_path="/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_I2I_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_T2T_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACONV_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LADD_train.json" \
--batch_size=16 \
--num_epoch=6 \
--external_save_dir=./checkpoints \
--save_hf_model \
--workers=8 \
--cross_attn_every_n_layers=4 \
--lr_schedule=cosine \
--learning_rate=1e-5 \
--warmup_steps_ratio=0.01

The error I got is
Traceback (most recent call last):
  File "/path/to/Otter/pipeline/train/instruction_following.py", line 498, in <module>
    main()
  File "/path/to/Otter/pipeline/train/instruction_following.py", line 451, in main
    model, optimizer, lr_scheduler, mimicit_loaders = accelerator.prepare(model, optimizer, lr_scheduler, mimicit_loaders)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1182, in prepare
    result = tuple(
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1183, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1022, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1311, in prepare_model
    torch.autocast(device_type=self.device.type, dtype=torch.bfloat16)(model.forward.__func__), model
AttributeError: 'function' object has no attribute '__func__'

Could you please give me some suggestions? I am very new to the accelerate package.
The GPU I used is A100 and I think it is not the GPU isssue.
Thanks!

I also encountered this problem and updating the accelerate package to the GitHub version worked for me (see this PR https://github.com/huggingface/accelerate/pull/1637).

Hi Bo, I tried the following script, but I got an error. I want to train the model using LA dataset and I added the image_processor and train_num_samples back to the args. The command I used is
accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml
 pipeline/train/instruction_following.py \
--pretrained_model_name_or_path=/path/to/OTTER-9B-INIT \
--mimicit_path="/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_I2I_instructions.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_T2T_instructions.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACONV_instructions.json,/path/toOtter/OneDrive_1_6-27-2023/LA/LADD_instructions.json" \
--images_path="/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json" \
--train_config_path="/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_I2I_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_T2T_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACONV_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LADD_train.json" \
--batch_size=16 \
--num_epoch=6 \
--external_save_dir=./checkpoints \
--save_hf_model \
--workers=8 \
--cross_attn_every_n_layers=4 \
--lr_schedule=cosine \
--learning_rate=1e-5 \
--warmup_steps_ratio=0.01

The error I got is
Traceback (most recent call last):
  File "/path/to/Otter/pipeline/train/instruction_following.py", line 498, in <module>
    main()
  File "/path/to/Otter/pipeline/train/instruction_following.py", line 451, in main
    model, optimizer, lr_scheduler, mimicit_loaders = accelerator.prepare(model, optimizer, lr_scheduler, mimicit_loaders)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1182, in prepare
    result = tuple(
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1183, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1022, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1311, in prepare_model
    torch.autocast(device_type=self.device.type, dtype=torch.bfloat16)(model.forward.__func__), model
AttributeError: 'function' object has no attribute '__func__'

Could you please give me some suggestions? I am very new to the accelerate package.
The GPU I used is A100 and I think it is not the GPU isssue.
Thanks!

I also meet this problem, anyone could offer me a hand?

Hi Bo, I tried the following script, but I got an error. I want to train the model using LA dataset and I added the image_processor and train_num_samples back to the args. The command I used is
accelerate launch --config_file=./pipeline/accelerate_configs/accelerate_config_fsdp.yaml
 pipeline/train/instruction_following.py \
--pretrained_model_name_or_path=/path/to/OTTER-9B-INIT \
--mimicit_path="/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_I2I_instructions.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_T2T_instructions.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACONV_instructions.json,/path/toOtter/OneDrive_1_6-27-2023/LA/LADD_instructions.json" \
--images_path="/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json,/path/to/Otter/mimic-it/convert-it/output/LA.json" \
--train_config_path="/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_I2I_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACR_T2T_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LACONV_train.json,/path/to/Otter/OneDrive_1_6-27-2023/LA/LADD_train.json" \
--batch_size=16 \
--num_epoch=6 \
--external_save_dir=./checkpoints \
--save_hf_model \
--workers=8 \
--cross_attn_every_n_layers=4 \
--lr_schedule=cosine \
--learning_rate=1e-5 \
--warmup_steps_ratio=0.01

The error I got is
Traceback (most recent call last):
  File "/path/to/Otter/pipeline/train/instruction_following.py", line 498, in <module>
    main()
  File "/path/to/Otter/pipeline/train/instruction_following.py", line 451, in main
    model, optimizer, lr_scheduler, mimicit_loaders = accelerator.prepare(model, optimizer, lr_scheduler, mimicit_loaders)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1182, in prepare
    result = tuple(
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1183, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1022, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/path/to/miniconda/envs/otter/lib/python3.9/site-packages/accelerate/accelerator.py", line 1311, in prepare_model
    torch.autocast(device_type=self.device.type, dtype=torch.bfloat16)(model.forward.__func__), model
AttributeError: 'function' object has no attribute '__func__'

Could you please give me some suggestions? I am very new to the accelerate package.
The GPU I used is A100 and I think it is not the GPU isssue.
Thanks!

I also meet this problem, anyone could offer me a hand?

See the above post at the first please.

We put our configs here.
You can use 1 GPU with at least 33-34G memory to start training. If you load the model with bf16 or fp16, you can lower the GPU requirement to 16-20G I think. But we didnt try to train a model under bf16 or fp16 (we actually use mixed precision).
You can refer the following code to load a model under certain precisions.
import os
import torch
import argparse
from .configuration_flamingo import FlamingoConfig
from .modeling_flamingo import FlamingoForConditionalGeneration

parser = argparse.ArgumentParser(description="Load model with precision")
parser.add_argument('--load_bit', type=str, choices=['fp16', 'bf16'], required=True, help="Choose either 'fp16' or 'bf16'")
args = parser.parse_args()

load_bit = args.load_bit

if load_bit == "fp16":
    precision = {"torch_dtype": torch.float16}
elif load_bit == "bf16":
    precision = {"torch_dtype": torch.bfloat16}

checkpoint_path = f"luodian/OTTER-9B-INIT"
model = FlamingoForConditionalGeneration.from_pretrained(checkpoint_path, device_map="auto", **precision)

# save model
checkpoint_path = checkpoint_path + f"_{load_bit}"
model.save_pretrained(checkpoint_path)

Thanks very much! I've implement it and trained on fp16, also change the data into torch.float16, but meet the error below:

Does anyone figure it out?


error on exporting note with "pre" tags

[x] I have verified in the Zotero addons screen that I have the latest release of BBT (https://github.com/retorquere/zotero-better-bibtex/releases/latest) and Zotero installed
[x] I am posting a single question, bug or feature request
[x] I have included a descriptive subject of the problem
[x] I am available for follow-up questions and testing
[x] If I am posting an export or key-generation related bug report, I have included an error-report by selecting the problematic reference(s), right-clicking, and submitting an BBT error report from that popup menu

Bug classification
error report reference: 5X7R8XDH-euc
I am trying to export my library as Better BibTex (which used to work fine in the past) and now I am getting an error point to javascript common.js.

This issue comes up with that library on a Windows as well as a Linux installation.

It only happens when exporting the URL in an note field. When exporting to a url field, it works.

The entry that caused the issue was set as Computer Program which if I remember correctly was generated with Zotero Connector from a Github page like one year ago. Maybe I also changed it manually afterwards but I can't remember..
I changed it to type Web Site and it works.
error in Terminal:
Possibly unhandled rejection:
duplicate field 'howpublished' for UsrsctpPortableSCTP2017
add@Better BibTeX:11872:23
["./Better BibTeX.ts"]/Translator.doExport@Better BibTeX:10381:17
doExport@Better BibTeX:132:5

:robot: this is your friendly neighborhood build bot announcing test build <IP_ADDRESS>52 ("actually fixes #1092")
Install in Zotero by downloading test build <IP_ADDRESS>52, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

Can you try 3052?

It looks like I can't reproduce the issue even with the previous version. I've already deleted/changed the misbehaving entries.

Ok. I reproduced it, updated to 3052 and the issue persists: new error report
B94J8QBZ-euc

:robot: this is your friendly neighborhood build bot announcing test build <IP_ADDRESS>55 ("actually fixes #1092")
Install in Zotero by downloading test build <IP_ADDRESS>55, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

You actually found a second problem; the first problem was reproducible with 5X7R8XDH-euc.

:robot: this is your friendly neighborhood build bot announcing test build <IP_ADDRESS>56 ("fixes 2nd report in #1092")
Install in Zotero by downloading test build <IP_ADDRESS>56, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

Alright, try 3056.

Only I can close issues; I like to keep them open until I've made sure the fix makes it into a release.

Thanks for the confirmation, I'll cut a new release to include this fix. You will be upgraded automatically from the version you have now.


PHPLIB-930: Update expected error in fle2-InsertFind-Unindexed test

https://jira.mongodb.org/browse/PHPLIB-930
Synced with mongodb/specifications@1a911b5e8911f6f57426ec1d451dbdc19480afd8

Please remove "https://jira.mongodb.org/browse/PHPLIB-930" from the commit message. I only drop that in the PR descriptions because GitHub sometimes fails to render the configured autolinking. The commit itself only needs the issue number as a prefix, though.
And friendly reminder to bump the status of PHPLIB-930 after opening a PR.

Diff looks fine now, but I see the following is now missing from the commit:
Synced with mongodb/specifications@1a911b5e8911f6f57426ec1d451dbdc19480afd8

Please make sure that ends up in the final commit when merging.


Contents and posts are not updating

After building and deployed to the vercel the contents and posts are not updating when I am creating new posts on WordPress. Do I need to build and deploy every time I create a new post?

hey @tareq0065 when using WordPress headlessly (and any other headless CMS) you need to make your hosting provider (Vercel) aware of the changes that you made, as there isn't any relationship between the 2 by default
you can use web hooks to do this by creating a Deploy Hook in Vercel:
https://vercel.com/docs/more/deploy-hooks
Then installing WP Webhooks:
https://wordpress.org/plugins/wp-webhooks/
To automatically trigger that hook on content update

@colbyfayock Hey this is awesome. Thanks for your reply ........

here's a tutorial that uses Netlify in case it helps https://spacejelly.dev/posts/how-to-automate-deployments-on-content-change-in-wordpress-with-wp-webhooks-and-netlify/


Lock file was specified instead of the locked directory.

The tests seems to fail because BioSamples have blocked access to the validate endpoint. This might be temporary or be due to the change from AAP to Webin.


Bug: system/Database/OCI8/Connection.php fails to parse valid DSN

PHP Version
8.2
CodeIgniter4 Version
4.5.2
CodeIgniter4 Installation Method
Composer (as dependency to an existing project)
Which operating systems have you tested for this bug?
Linux
Which server did you use?
apache
Database
Oracle
What happened?
Some valid DSN strings fail the isValidDSN() method, meaning that the $validDSNs['tsn'] regex is too strict in its check.
Steps to Reproduce
Try the DSN (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=TCP.world)(PROTOCOL=TCP)(Host=dbodsprod.is.colostate.edu)(Port=1521)))(CONNECT_DATA=(SERVICE_NAME=odsprod.infosys.colostate.edu)))
Expected Output
preg_match($validDSNs['tsn']) returns true which causes isValidDSN() to return true which causes Connection->connect() to return a a valid resource
Anything else?
No response

Hi, the $validDSNs['tsn'] regex has no problem.
See https://3v4l.org/h0Xt4
What's your exact issue?

Thanks @kenjis . I was debugging this incorrectly.
At line 123 of system/Database/OCI8/Connection.php if (empty($this->DSN) && ! $this->isValidDSN()) {
$this->DSN is NULL for me, which causes the preg_match to fail.
If I change line 108 to if (preg_match($regexp, $this->DSN??'')) { the connection works

It seems $this->DSN in  isValidDSN() is always an empty string '' or null.
The first if condition should be ! empty($this->DSN) && ! $this->isValidDSN() ?
https://github.com/codeigniter4/CodeIgniter4/blob/53bfdef9f2b0370d8ca49ef068778a6839a8ed22/system/Database/OCI8/Connection.php#L123-L125
https://github.com/codeigniter4/CodeIgniter4/blob/53bfdef9f2b0370d8ca49ef068778a6839a8ed22/system/Database/OCI8/Connection.php#L105-L114

@jdpurdyvi How do you configure Config\Database?

The database connection is configured using a .env file. There is no line for database.xyz.DSN.
I'm not sure that changing line 123 solves the issue. I tried that change and got an exception of Main connection [OCI8]: oci_connect(): ORA-12162: TNS:net service name is incorrectly specified. I think the call to buildDSN() is still needed... 🫠

At first, don't use .env if you cannot connect the database.
After debugging, use .env.
See also
https://github.com/codeigniter4/CodeIgniter4/blob/53bfdef9f2b0370d8ca49ef068778a6839a8ed22/app/Config/Database.php#L129-L156

Thanks. I will adjust my database connection setup.
In the meantime, if you can assume that "$this->DSN in isValidDSN() is always an empty string '' or null." and we know that preg_match does not accept null as the 2nd parameter, why not check for null with the null coalesce operator?

@jdpurdyvi Try #8986

Closed by #8986

Apologies for the delay. https://github.com/codeigniter4/CodeIgniter4/pull/8986 works in my env. Thanks!


DNS configuration

Is your feature request related to a problem? Please describe.
You can't configure any aspect of client DNS (servers, timeouts...)
Describe the solution you'd like
To ba able to configure at least DNS to use to resolve names
Describe alternatives you've considered
...
Additional context
I've dived into Pulsar client DNS internal configuration and I didn't found any way to properly configure DNS client properties on my system. Moreover if "autodiscovery" cannot find any dns it will attempt to use Google ones! In a corporate environment this isn't acceptable.
I could write you a patch to add dns configuration but I'm blocked between 3 alternatives:

add every needed dns configuration parameter on client configuration (... with some configuration bloating)
just require to provide an already configured netty dns and use current one as fallback (but it couldn't be configured via properties but only programmatically and expose internal netty dependences)
create a new "DNSClient" interface and a netty implementation and require to provide such dns object (but still cannot be configured via properties...)

2 seems to be a reasonable approach to me

Created PR for (2)

Closed as stale. See https://github.com/apache/pulsar/pull/6382#issuecomment-1340126622.


Fix state machine for opensent/openconfirm

Currently we make the other peer send the first OPEN message, but in practice we need to either send an OPEN message when the connection starts, or start a timer that eventually initiates with an OPEN method. Both of these would then move to the OpenSent stage which hasn't yet been defined.
The current behaviour is as if we have an OpenDelayTimer set with a super high value which isn't necessarily wrong, but isn't the right intention (and makes it harder for us to build active peering)

Remember this is different from passive vs active connection - that just decides whether you're trying to open a TCP connection. The OpenDelayTimer login happens regardless of who initiated the TCP connection.


ci: increasing the default timeout setting for pip install

Related issue number
Fixes https://github.com/alibaba/GraphScope/actions/runs/6505103462/job/17668168357

Codecov Report

Merging #3292 (261d4c2) into main (c8a387c) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3292   +/-   ##
=======================================
  Coverage   42.02%   42.02%
=======================================
  Files         101      101
  Lines       11013    11013
=======================================
  Hits         4628     4628
  Misses       6385     6385

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8a387c...261d4c2. Read the comment docs.


TypeError: Cannot read property 'url' of null at cacheCurrentPage

I have a form with 4 tabs, each with history, when changed the url adds #tab1 ... #tab4
I append on "ajax:before" a remote=true form a hidden field with the anchor and send it back like this:
My Controller
...
format.js { redirect_to XX_path(YYY, anchor: params[:anchor]), change: 'edit-campaign-actions' }
...

If I load the edit form at #tab2 and I edit a field there, it will redirect me back to the same #tab2 and everything goes ok.
When I load the edit form in #tab2 move to #tab3 and the redirect goes to #tab3, I get this:
TypeError: Cannot read property 'url' of null
    at cacheCurrentPage (http://www.audify.io:2026/assets/dashboard.js:32256:52)
    at Object.fetch (http://www.audify.io:2026/assets/dashboard.js:32139:5)
    at eval (eval at <anonymous> (http://www.audify.io:2026/assets/dashboard.js:1746:22), <anonymous>:1:12)
    at eval (native)
    at http://www.audify.io:2026/assets/dashboard.js:1746:22
    at Function.jQuery.extend.globalEval (http://www.audify.io:2026/assets/dashboard.js:1747:7)
    at jQuery.ajaxSetup.converters.text script (http://www.audify.io:2026/assets/dashboard.js:11146:11)
    at ajaxConvert (http://www.audify.io:2026/assets/dashboard.js:10216:19)
    at done (http://www.audify.io:2026/assets/dashboard.js:10633:15)
    at XMLHttpRequest.jQuery.ajaxTransport.send.callback (http://www.audify.io:2026/assets/dashboard.js:11093:8)

Forgot to add I partially solved by forking and returning
return if !cacheCurrentPage

Do you have a live website where we can replicate the bug?

@Thibaut sadly not. I was using the master this repo as gem. Ask me for any info that might help.

@imton does this occur in both Chrome and Firefox?

Also, make sure you don't have code that calls stopImmediatePropagation in a hashchange event handler.

Haven't tested it on Firefox.
Well I am using https://github.com/aidanlister/jquery-stickytabs
Gaston Morixe
Partner Black.uy
+59898704249
@_imton http://twitter.com/_imton
Montevideo, Uruguay
On Sun, May 17, 2015 at 11:00 AM, Thibaut Courouble <
<EMAIL_ADDRESS>wrote:

Also, make sure you don't have code that calls stopImmediatePropagation
in a hashchange event handler.
—
Reply to this email directly or view it on GitHub
https://github.com/rails/turbolinks/issues/530#issuecomment-102806223.

I tried my best to replicate using jquery-stickytabs, but wasn't able to.
Feel free to re-open if you can provide a failing test case or example code that we can use to replicate.


[SPARK-31350][SQL] Coalesce bucketed tables for sort merge join if applicable

What changes were proposed in this pull request?

When two bucketed tables with different number of buckets are joined, it can introduce a full shuffle:
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "0")
val df1 = (0 until 20).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k")
val df2 = (0 until 20).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k")
df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t2")
val t1 = spark.table("t1")
val t2 = spark.table("t2")
val joined = t1.join(t2, t1("i") === t2("i"))
joined.explain

== Physical Plan ==
*(5) SortMergeJoin [i#44], [i#50], Inner
:- *(2) Sort [i#44 ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(i#44, 200), true, [id=#105]
:     +- *(1) Project [i#44, j#45, k#46]
:        +- *(1) Filter isnotnull(i#44)
:           +- *(1) ColumnarToRow
:              +- FileScan parquet default.t1[i#44,j#45,k#46] Batched: true, DataFilters: [isnotnull(i#44)], Format: Parquet, Location: InMemoryFileIndex[...], PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int,j:int,k:string>, SelectedBucketsCount: 8 out of 8
+- *(4) Sort [i#50 ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(i#50, 200), true, [id=#115]
      +- *(3) Project [i#50, j#51, k#52]
         +- *(3) Filter isnotnull(i#50)
            +- *(3) ColumnarToRow
               +- FileScan parquet default.t2[i#50,j#51,k#52] Batched: true, DataFilters: [isnotnull(i#50)], Format: Parquet, Location: InMemoryFileIndex[...], PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int,j:int,k:string>, SelectedBucketsCount: 4 out of 4

This PR proposes to introduce coalescing buckets when the following conditions are met to eliminate the full shuffle:

Join is the sort merge one (which is created only for equi-join).
Join keys match with output partition expressions on their respective sides.
The larger bucket number is divisible by the smaller bucket number.
spark.sql.bucketing.coalesceBucketsInSortMergeJoin.enabled is set to true.
The ratio of the number of buckets should be less than the value set in spark.sql.bucketing.coalesceBucketsInSortMergeJoin.maxBucketRatio.

Why are the changes needed?

Eliminating the full shuffle can benefit for scenarios where two large tables are joined. Especially when the tables are already bucketed but differ in the number of buckets, we could take advantage of it.
Does this PR introduce any user-facing change?

If the bucket coalescing conditions explained above are met, a full shuffle can be eliminated (also note that you will see SelectedBucketsCount: 8 out of 8 (Coalesced to 4) in the physical plan):
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "0")
spark.conf.set("spark.sql.bucketing.coalesceBucketsInSortMergeJoin.enabled", "true")
val df1 = (0 until 20).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k")
val df2 = (0 until 20).map(i => (i % 7, i % 11, i.toString)).toDF("i", "j", "k")
df1.write.format("parquet").bucketBy(8, "i").saveAsTable("t1")
df2.write.format("parquet").bucketBy(4, "i").saveAsTable("t2")
val t1 = spark.table("t1")
val t2 = spark.table("t2")
val joined = t1.join(t2, t1("i") === t2("i"))
joined.explain

== Physical Plan ==
*(3) SortMergeJoin [i#44], [i#50], Inner
:- *(1) Sort [i#44 ASC NULLS FIRST], false, 0
:  +- *(1) Project [i#44, j#45, k#46]
:     +- *(1) Filter isnotnull(i#44)
:        +- *(1) ColumnarToRow
:           +- FileScan parquet default.t1[i#44,j#45,k#46] Batched: true, DataFilters: [isnotnull(i#44)], Format: Parquet, Location: InMemoryFileIndex[...], PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int,j:int,k:string>, SelectedBucketsCount: 8 out of 8 (Coalesced to 4)
+- *(2) Sort [i#50 ASC NULLS FIRST], false, 0
   +- *(2) Project [i#50, j#51, k#52]
      +- *(2) Filter isnotnull(i#50)
         +- *(2) ColumnarToRow
            +- FileScan parquet default.t2[i#50,j#51,k#52] Batched: true, DataFilters: [isnotnull(i#50)], Format: Parquet, Location: InMemoryFileIndex[...], PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int,j:int,k:string>, SelectedBucketsCount: 4 out of 4

How was this patch tested?

Added unit tests

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25520/
Test PASSed.

Test build #120821 has started for PR 28123 at commit 94720e0.

@cloud-fan @maropu @gatorsmile Could you check if this PR is on the right path? Thanks in advance!

Test build #120822 has started for PR 28123 at commit a15af6b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25521/
Test PASSed.

Test build #120822 has finished for PR 28123 at commit a15af6b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120822/
Test PASSed.

Test build #120821 has finished for PR 28123 at commit 94720e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120821/
Test PASSed.

This optimization always works well? e.g., 100000 buckets vs 2 buckets.

This optimization always works well? e.g., 100000 buckets vs 2 buckets.

Not always, so this is enabled if spark.sql.bucketing.coalesce is set to true.

This optimization always works well? e.g., 100000 buckets vs 2 buckets.

Not always, so this is enabled if spark.sql.bucketing.coalesce is set to true.

If so, I think we need a threshold config to turn of this optimization if the numbers of joined buckets have a large gap.

also cc: @viirya

Test build #121238 has started for PR 28123 at commit 40b0707.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25925/
Test PASSed.

Test build #121238 has finished for PR 28123 at commit 40b0707.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121238/
Test FAILed.

Test build #121554 has started for PR 28123 at commit 0b9abee.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26236/
Test PASSed.

Test build #121555 has started for PR 28123 at commit c93322b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26237/
Test PASSed.

Test build #121557 has started for PR 28123 at commit 347c2de.

Test build #121557 has finished for PR 28123 at commit 347c2de.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121557/
Test FAILed.

Test build #121555 has finished for PR 28123 at commit c93322b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121555/
Test FAILed.

Test build #121554 has finished for PR 28123 at commit 0b9abee.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121554/
Test FAILed.

Test build #121715 has started for PR 28123 at commit c78e2a2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26395/
Test PASSed.

Test build #121715 has finished for PR 28123 at commit c78e2a2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121715/
Test FAILed.

Test build #121787 has started for PR 28123 at commit 26cfb63.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26463/
Test PASSed.

Test build #121787 has finished for PR 28123 at commit 26cfb63.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121787/
Test FAILed.

Test build #121812 has started for PR 28123 at commit ebe062f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26487/
Test PASSed.

Test build #121812 has finished for PR 28123 at commit ebe062f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121812/
Test FAILed.

retest this please

Test build #121821 has started for PR 28123 at commit ebe062f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26495/
Test PASSed.

Test build #121821 has finished for PR 28123 at commit ebe062f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121821/
Test PASSed.

Test build #122032 has started for PR 28123 at commit 19b8cd8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26705/
Test PASSed.

Thanks for the updates! Also, could you update the PR description, too?

Test build #122032 has finished for PR 28123 at commit 19b8cd8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122032/
Test FAILed.

Test build #122102 has started for PR 28123 at commit 148a6fc.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26776/
Test PASSed.

Test build #122102 has finished for PR 28123 at commit 148a6fc.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122102/
Test FAILed.

retest this please

Test build #122118 has started for PR 28123 at commit 148a6fc.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26792/
Test PASSed.

Test build #122118 has finished for PR 28123 at commit 148a6fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122118/
Test PASSed.

Test build #122297 has started for PR 28123 at commit eeb0ec7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26967/
Test PASSed.

Test build #122297 has finished for PR 28123 at commit eeb0ec7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122297/
Test FAILed.

retest this please

Test build #122311 has started for PR 28123 at commit eeb0ec7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26981/
Test PASSed.

Test build #122311 has finished for PR 28123 at commit eeb0ec7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122311/
Test PASSed.

@maropu Any more comments? Thanks!

To move forward further, I think we need more comments from other reviewers. kingly ping @cloud-fan @viirya

@cloud-fan Do you think this PR can move forward? Thanks!

The general idea LGTM. Two high-level questions ：

It's not possible to have an accurate cost model to guide this optimization. Do we have a heuristic? like coalescing 10000 buckets to 2 is very likely to cause regression as parallelism is reduced too much.
Are we going to support joins with more than 2 tables? e.g. "100 buckets table" join "50 buckets table" join "10 buckets table".

It's not possible to have an accurate cost model to guide this optimization. Do we have a heuristic? like coalescing 10000 buckets to 2 is very likely to cause regression as parallelism is reduced too much.

A heuristic is discussed here to set the default value for spark.sql.bucketing.coalesceBucketsInJoin.maxNumBucketsDiff. And the config should help preventing the regression by the reduced parallelism.

Are we going to support joins with more than 2 tables? e.g. "100 buckets table" join "50 buckets table" join "10 buckets table".

Good idea. I can do a follow-up PR to improve the current rule (which doesn't support nested joins)?

Test build #123096 has started for PR 28123 at commit c1e0d26.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27730/
Test PASSed.

Test build #123096 has finished for PR 28123 at commit c1e0d26.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123096/
Test PASSed.

Could you update the PR description, too, based on the current PR approach? e.g., Join is the equi-join. -> Join is the sort merge one with equality conditions

Test build #123321 has started for PR 28123 at commit e54d34b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27943/
Test PASSed.

Test build #123322 has started for PR 28123 at commit eee4884.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27944/
Test PASSed.

Test build #123322 has finished for PR 28123 at commit eee4884.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class CoalesceBucketsInSortMergeJoin(conf: SQLConf) extends Rule[SparkPlan]

Test build #123321 has finished for PR 28123 at commit e54d34b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123322/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123321/
Test PASSed.

Test build #123349 has started for PR 28123 at commit ed2d316.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27970/
Test PASSed.

Test build #123349 has finished for PR 28123 at commit ed2d316.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123349/
Test PASSed.

it will be good to see benchmark numbers of a typical bucket join that can benefit from this patch.

Test build #123777 has finished for PR 28123 at commit 1df3b9f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123777/
Test PASSed.

The patch itself LGTM, look forward to seeing the numbers!

Here are some numbers when I joined two tables (store_sales from TPC-DS - 100 SF) and did count on it. It's run on 8 executors (8 cores each) and generates about 47GB of shuffle.

Bucket Size #1
Bucket Size #2
Coalesce On (ms)
Coalesce Off (ms)
Gain (%)

512
256
40495
48435
16.39310416

512
128
42459
49597
14.39199952

512
64
45760
48888
6.398298151

256
128
41241
49034
15.8930538

256
64
42902
51063
15.98221804

128
64
44131
53192
17.03451647

There is a modest 15% gain (for ratio up to 4), WDYT?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28506/
Test PASSed.

Test build #123883 has started for PR 28123 at commit cf5b835.

Test build #123883 has finished for PR 28123 at commit cf5b835.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123883/
Test FAILed.

retest this please

Test build #123899 has started for PR 28123 at commit cf5b835.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28522/
Test PASSed.

Test build #123899 has finished for PR 28123 at commit cf5b835.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

retest this please

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123899/
Test FAILed.

Test build #123926 has started for PR 28123 at commit cf5b835.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28548/
Test PASSed.

Test build #123926 has finished for PR 28123 at commit cf5b835.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123926/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28566/
Test PASSed.

Test build #123945 has started for PR 28123 at commit cf5b835.

Test build #123945 has finished for PR 28123 at commit cf5b835.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123945/
Test FAILed.

retest this please

Test build #123962 has started for PR 28123 at commit cf5b835.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28584/
Test PASSed.

Test build #123962 has finished for PR 28123 at commit cf5b835.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123962/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28585/
Test PASSed.

Test build #123963 has started for PR 28123 at commit cf5b835.

Test build #123963 has finished for PR 28123 at commit cf5b835.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123963/
Test FAILed.

retest this please

Test build #123967 has started for PR 28123 at commit cf5b835.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28590/
Test PASSed.

Test build #123967 has finished for PR 28123 at commit cf5b835.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123967/
Test PASSed.

Test build #124194 has started for PR 28123 at commit e231268.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28811/
Test PASSed.

Test build #124194 has finished for PR 28123 at commit e231268.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124194/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28821/
Test PASSed.

Test build #124205 has started for PR 28123 at commit e231268.

Test build #124205 has finished for PR 28123 at commit e231268.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124205/
Test PASSed.

Test build #124229 has started for PR 28123 at commit 38c7d6e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28847/
Test PASSed.

Test build #124229 has finished for PR 28123 at commit 38c7d6e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124229/
Test PASSed.

Test build #124258 has started for PR 28123 at commit 62a04a3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28876/
Test PASSed.

Test build #124258 has finished for PR 28123 at commit 62a04a3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124258/
Test FAILed.

retest this please

Test build #124271 has started for PR 28123 at commit 62a04a3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28888/
Test PASSed.

Test build #124271 has finished for PR 28123 at commit 62a04a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124271/
Test PASSed.

Thanks! I appreciate your hard work, @imback82 ! Merged to master. Also, thanks for the reviews, @cloud-fan @viirya !

Thanks @imback82 for making this change!
Sorry for late comment, just a few questions:
(1).Is there a reason why we don't cover ShuffledHashJoin as well? (we are seeing in production, people also use ShuffledHashJoin a lot for joining bucketed tables when one side is small)
(2).Per this PR, the ordering property of coalesced bucket files does not preserve (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L317), and the ordering can be preserved through a sort-merge-way read of all sorted buckets file. This can help when reading multiple partitions of bucketed table.
(3).We are seeing in production, coalescing might hurt the parallelism, if the number of buckets are too few. Another way to avoid shuffle and sort, is to split/divide the table with less buckets. E.g. joining tables with t1 (8 buckets) and t2 (32 buckets), we can keep number of tasks to be 32, and each task for reading t1 table will have a filter at run-time to only keep its portion of table (divide the table with less buckets). This has downside of reading the t1 more than once from multiple tasks, but if the size of t1 is not big, it's a good trade off to have more parallelism (and may be better than shuffling t1 directly).
We are running above 3 features years in facebook (https://databricks.com/session_eu19/spark-sql-bucketing-at-facebook), and I would like to make or help the followup change if this sounds reasonable for everyone. cc @imback82, @cloud-fan, @viirya, @gatorsmile and @sameeragarwal.

Thanks for your interest, @c21

(1).Is there a reason why we don't cover ShuffledHashJoin as well? (we are seeing in production, people also use ShuffledHashJoin a lot for joining bucketed tables when one side is small)

As you said in (3), too, I think that's because there is a concern where coalescing might hurt the parallelism. You can see the related discussion in the history: https://github.com/apache/spark/pull/28123#discussion_r427073319
As for (1) and (3), IMO its worth digging into it for more improvements.

(2).Per this PR, the ordering property of coalesced bucket files does not preserve (https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L317), and the ordering can be preserved through a sort-merge-way read of all sorted buckets file. This can help when reading multiple partitions of bucketed table.

I think that's the long-standing issue we have. Have you checked the discussion in SPARK-24528? If you're interested in the issue, you can revisit it there.

(3).We are seeing in production, coalescing might hurt the parallelism, if the number of buckets are too few. Another way to avoid shuffle and sort, is to split/divide the table with less buckets. E.g. joining tables with t1 (8 buckets) and t2 (32 buckets), we can keep number of tasks to be 32, and each task for reading t1 table will have a filter at run-time to only keep its portion of table (divide the table with less buckets). This has downside of reading the t1 more than once from multiple tasks, but if the size of t1 is not big, it's a good trade off to have more parallelism (and may be better than shuffling t1 directly).

I had a rough POC few months back, where each row is filtered out based on its bucket id, but I never got a chance to run benchmarks. @c21 Do you have some numbers to share? I am wondering how reading multiple copies impacts the overall runtime.

Thank you @maropu and @imback82

As for (1) and (3), IMO its worth digging into it for more improvements.

For (1): I created a PR to cover shuffled hash join as well - https://github.com/apache/spark/pull/29079. Could you help take a look? Thanks.

I had a rough POC few months back, where each row is filtered out based on its bucket id, but I never got a chance to run benchmarks. @c21 Do you have some numbers to share? I am wondering how reading multiple copies impacts the overall runtime.

We have some internal numbers in production, e.g. we are seeing like 50% CPU improvement for specific queries. So comparing divide (extra N times for reading input table, N is the ratio between two tables buckets) and non-divide (extra 1 time for shuffle write, and 1 time for read), if the join big tables buckets ratio is 2 (e.g. join 510 buckets with 1024 buckets), dividing can be better than non-dividing as we only do 1 extra read for input table, but avoid 1 shuffle write and 1 shuffle read (but this also depends on efficiency of shuffled service). I think we can add benchmark in JoinBenchmark for joining table with buckets ratio to be 2 to showcase this improvement.
Re POC - I feel overall approach looks good to me. But IMO I think we should do the coalesce/divide in physical plan rule, but not logical plan rule. Also I think we probably can leave vectorization code path aside now, as it introduces too much change to handle vectorization when doing filter for it. Let me know if you are still on it, or I can help with this feature as well. Thanks.

Thanks @c21!

Re POC - I feel overall approach looks good to me. But IMO I think we should do the coalesce/divide in physical plan rule, but not logical plan rule.

Yes, I agree. (The POC was done before this PR, which also started with logical plan but changed to physical plan rule per suggestion)

Also I think we probably can leave vectorization code path aside now, as it introduces too much change to handle vectorization when doing filter for it.

Without supporting ColumnarBatch, you cannot enable the wholestage codegen. Is that desirable?

Let me know if you are still on it, or I can help with this feature as well. Thanks.

I will clean up the POC and create a PR with some benchmark numbers.

Here are some numbers when I joined two tables (store_sales from TPC-DS - 100 SF) and did count on it. It's run on 8 executors (8 cores each) and generates about 47GB of shuffle.
Bucket Size #1	Bucket Size #2	Coalesce On (ms)	Coalesce Off (ms)	Gain (%)
512	256	40495	48435	16.39310416
512	128	42459	49597	14.39199952
512	64	45760	48888	6.398298151
256	128	41241	49034	15.8930538
256	64	42902	51063	15.98221804
128	64	44131	53192	17.03451647
There is a modest 15% gain (for ratio up to 4), WDYT?

@imback82 sounds good. I am refactoring and to making the coalesce cover shuffled hash join as well in (https://github.com/apache/spark/pull/29079).
I would like to run same query as you did here, and verified performance numbers. Wondering could you share more instructions for which query in TPCDS you ran, or the query you used if you changed based the benchmark? Thanks.

Without supporting ColumnarBatch, you cannot enable the wholestage codegen

Sorry I might miss something, but I feel it's not true. From my understanding, wholestage codegen is on row-based, cannot work with vectorization yet.


Respect SvelteKit’s trailingSlash = 'always' behavior

Discussed in https://github.com/jycouet/kitql/discussions/693

Originally posted by danieljb August  4, 2024
Hi, I’ve set up kitql in a project configured to use trailing slashes via export in a +layout.ts file.
This generally works, but the links generated by kitql are missing the trailing slash. This can lead to an additional request when SvelteKit redirects to the with-trailing-slash url path, e.g. a link to /about will be redirected to /about/.
I tried adding the trailing slash to kitql’s route call, e.g. route('/about/') but this results in a type error. Is there a way to configure kitql to accept the trailing slash behavior?

A global config flag would definitely work in my case. Fully agree this would be a good first step.

Hi @danieljb
It should be done in<EMAIL_ADDRESS>please re open if needed.
Let me know

Amazing, thanks a lot for the fast response and implementation. Very excited to check it out. Will let you know if anything comes up.


Update logo + link

See https://github.com/cert-manager/csi-driver/pull/80 for justification on logo changes

/test pull-cert-manager-trust-verify

/test pull-cert-manager-trust-smoke

(added an LGTM manually because this is so trivial it doesn't seem worth pestering people about)


Update due to security alerts on dependencies

RE-CHECK

RE-CHECK!

RE-CHECK!

Automated adapter checker
ioBroker.sainlogic

[ ] :heavy_exclamation_mark: [E301] Tests on Travis-ci.com are broken. Please fix.
[ ] :heavy_exclamation_mark: [E701] No actual year found in LICENSE. Please add "Copyright (c) 2021 Fogg<EMAIL_ADDRESS>at the start of LICENSE

Add comment "RE-CHECK!" to start check anew

RE-CHECK!

Automated adapter checker
ioBroker.sainlogic

:thumbsup: No errors found
Add comment "RE-CHECK!" to start check anew

Here are some review comments

Please use iobroker translation tools to add all needed languages for the relevant fields in io-package
Are callbacks available here ? https://github.com/phifogg/ioBroker.sainlogic/blob/master/main.js#L277-L280 that call when really anything is done? especially for the listener case? If yes please use that to call the unload callback
travis only executes the package testing and no adapter testing
Thank you for checking and fixing

Thanks. Some questions back:
-> Translate: I found https://weblate.iobroker.net/ - is that what you mean? and if, where is a how-to what I need to do? Frankly, I believe german and english are enough. Have not seen anything else beeing requested by users.
-> Callbacks: Not understanding your question here. Listener is a service which opens a port and waits on inbound connection. On termination of the adapter I need to gracefully close/shutdown. This is what is happening here.
-> Correct. What else do I need here? Simulating inbound http calls to update state values?
I understand a certain level of quality gate, but we are all doing this as hobby and it feeld like a lot of burdon you put on a adapter developers. Just my 5 cent.

@phifogg
For translate: The basic information in io-package are required to exist in alle languages because ioBroker offers these and we have a user base in these languages already. The mentioned tool is referenced in https://github.com/ioBroker/ioBroker.repositories#development-and-coding-best-practices ... --> https://translator.iobroker.in/ past text, get translation object back.
For Callbacks: Many "network based servers" provide callbacks or resolved promises  that are cakled after the port is actually really closed (in many cases close is still an async action). If your library/what you use provides sucha close-callback then preferrable use those to call the unload callback. iobroekr assumes that any resources are freed up when you call the unload-callback - so when the ports needs another second to be really closed it can give errors on adapter restarts or in compactmode. Please consider the async nature of JavaScript!
For Testing: You should at least also run the "Integration" tests in CI to see that adapter can start. If you like you can add more testing like simulation http calls, but thats optional.
I understand your feedback on "the burdon", but I think what we do here is bare minimal and we need to make sure that Adapters in the repository are not "hurting" users system and compact mode as example requires some care. Also each Adapter has it's part on how users see the whole ioBroker ecosystem in quality and such, so it is a fine balance we try to reach here.

@Apollon77
I have added translations for the UI parts now and checked on the callback when closing my http server. These should be adressed now.
As for the integration test in the CI pipeline, i'll check into that with the next release. Hope that will do.

RE-CHECK!

Automated adapter checker
ioBroker.sainlogic

[ ] :heavy_exclamation_mark: [E506] Word "" is not translated to german in admin/words.js. You can use https://translator.iobroker.in/ for translations
[ ] :heavy_exclamation_mark: [E506] Word "" is not translated to russian in admin/words.js. You can use https://translator.iobroker.in/ for translations

Add comment "RE-CHECK!" to start check anew

RE-CHECK!

Automated adapter checker
ioBroker.sainlogic

:thumbsup: No errors found
Add comment "RE-CHECK!" to start check anew


Faster builds using Parcel

Is your proposal related to a problem?
Parcel is a "Blazing fast, zero configuration web application bundler"
Based on a reasonably sized app, containing 1726 modules, 6.5M uncompressed. Built on a 2016 MacBook Pro with 4 physical CPUs.

Bundler
Time

browserify
22.98s

webpack
20.71s

parcel
9.98s

parcel - with cache
2.64s

Describe the solution you'd like
Use Parcel instead of webpack to build React app.
https://en.parceljs.org/recipes.html#react
https://en.parceljs.org/typeScript.html#when-using-react

A lot of parcel's functionality overlaps with create-react-app, in terms of providing similar zero-configuration functionalities.  Decreased bundle time probably varies on an app-by-app basis and it alone is not enough to justify spend tons of work porting it over at this point.  Webpack 5 is also around the corner which promises faster builds.


Support KHR_animation_pointer

The extension KHR_animation_pointer provides a path to the element to animate, and the path might be like this: /materials/0/pbrMetallicRoughness/baseColorFactor
Unfortunately, parsing such string is rather difficult in tinygltf without a direct access to JSON.
Would it be possible to provide a function, like  void* resolvePointerPath(const std::string& pointerPath)  which would return a pointer to the element?
This scene from the official glTF-Asset can be used for testing: https://github.com/KhronosGroup/glTF-Sample-Assets/tree/main/Models/AnimatedColorsCube

You can contribute writing a function to parse string in KHR_animation_pointer.
PR is much appreciated!


[discussion] task::block_on -> thread::block_on?

Okay so this might be a terrible idea, and I'm not convinced myself -- but I think it's still worth bringing up because there might be value to it.
In https://github.com/async-rs/async-std/pull/251 we're going to add task::blocking, but the name is kind of similar to task::block_on. The difference between the two is that one takes a closure that will block a thread, while the other one blocks the current thread:
fn block_on(fut: impl Future<Output = T>) -> T;
fn blocking(fut: impl Future<Output = T>) -> JoinHandle<T>;

What I'm slightly worried about is that people will not catch the difference between the two methods, and use block_on instead of blocking. And to be fair; this will probably work in most situations -- except performance will be terrible because this is not at all intended for this.
So instead I'm wondering if it would perhaps make sense to move block_on to be part of the thread submodule. It already has several other functions related to blocking the current thread, and this would be another one in that line.
// current
fn thread::park();
fn thread::park_timeout(dur: Duration);
fn thread::sleep(dur: Duration);
fn thread::yield_now();

// added
fn block_on(fut: impl Future<Output = T>) -> T;

A reason why I'm kind of wondering about using the thread submodule is because we probably want to give people access to the threadpool to configure it, and I feel thread may be a good option to put it (though I have no idea which knobs we'd want to allow people to turn). Which if we have that submodule anyway we may want to consider this direction also. I'm not sure though!
Perhaps another way to go about this would be to make block_on fail if called inside of a task context:
use crate::task::worker::get_task;

let task = get_task(|task| task.clone());
debug_assert!(task.is_none(), "`task::block_on()` called inside the context of a task. Did you mean to call `task::blocking?`");

Oh also; none of this is pressing at all, and just wanted to bring this up as something that may be interesting to consider. Thanks!

Okay, riffing on this a bit more. Now that #299 is a thing, things have changed even more as spawn_blocking takes a regular closure rather than an async block now:
fn task::spawn(impl Future) -> JoinHandle;
fn task::spawn_thread(impl FnOnce) -> JoinHandle;

So there seems to be somewhat of a parallel here now with thread. I think we could do something similar there:
fn thread::spawn(impl FnOnce) -> JoinHandle;
fn thread::spawn_async(impl Future) -> JoinHandle;

This creates a nice parallel I think, and nicely bridges between the two concepts. We could perhaps even take the names slightly further, and call them each other's submodule:
fn task::spawn(impl Future) -> task::JoinHandle;
fn task::spawn_thread(impl FnOnce) -> task::JoinHandle;

fn thread::spawn(impl FnOnce) -> thread::JoinHandle;
fn thread::spawn_task(impl Future) -> thread:: JoinHandle;

The only concern here is that spawn_thread might sound like it's a one-off thread, while we in fact do use a thread pool to back things up. But I'm not sure if that's more important than the FANTASTIC API SYMMETRY WE COULD HAVE!

Okay, tried to implement this and thread::spawn_task can't and shouldn't return a JoinHandle.
That kind of begs the question whether it still makes sense the way it's done (or whether it should have a different kind of join handle), but in general I still kind of like the direction we're proposing, even without the JoinHandle being returned.

I don't think this is worth it at this time. Closing!


Progress ring spins too fast

Current Behavior
Progress ring spins too fast
Expected Behavior
It should look like it does on the docs site
Steps To Reproduce
<script lang="ts">
    import { ProgressRing } from '@skeletonlabs/skeleton-svelte';
</script>

    <ProgressRing
        value={null}
        size="size-14"
        meterStroke="stroke-tertiary-600-400"
        trackStroke="stroke-tertiary-50-950"
    >
    </ProgressRing>

Link to Reproduction / Stackblitz
https://discord.com/channels/1003691521280856084/1191790107867488316/1307984229048647724

I think the issue is on your side, perhaps an extension or something?
It works fine when reproducing in stackblitz: https://www.sveltelab.dev/k1plh3x5eg1tml9

@liamwh yeah we'll need more information here. We use the stock spin animation provided by Tailwind itself.
https://github.com/skeletonlabs/skeleton/blob/next/packages/skeleton-svelte/src/lib/components/ProgressRing/ProgressRing.svelte#L49
This will need to be exposed as a style prop, but otherwise it's pretty straight forward how it works:

If there's a null value -> enter indeterminate mode
If indeterminate mode -> apply the spin class

Thanks for the swift response! I understand the API (very intuitive 👏), just not why mine is spinning so fast.
I'll take a look into it further when possible. Feel free to close and I can re-open if necessary.

Sure @liamwh if you can provide us steps to replicate we would be happy to take a look! As others pointed out on Discord, it could be related to a particular browser - so maybe cross check? But I certainly have seen that one come up yet!


Invoices are printed only as grayscale

Hello all.
I'm impressed with AdminLTE. It will be a great way to control some of my websites.
However, I have a problem. I tried to use the invoice example to make a "summary printer" but no matter what I do, the text is going to be printed always in grey-scale.
I searched the CSS code searching for filters but I can't found any related with the text to remove the colors.
Any of you can give me some guidance about how to solve it? I don't know if I'm doing something wrong or if it is a problem with the panel.
Thanks.

Hi,
Bootstrap Print CSS, print in black only.
check Bootstrap.css line 192 and 196
@media print {

Hi,
AdminLTE doesn't apply this kind of restriction. However, I see your problem and I suggest you consider asking the Bootstrap community since the restriction might be applied from their side.
You may want to review this question for a quick fix.


Coud not initialize dev

Which Category is your question related to?
Amplify CLI
Amplify CLI Version
4.46.1
What AWS Services are you utilizing?
AWS Amplify - JS
Provide additional details e.g. code snippets
I upgraded cli to version 4.46.1. I have created models using the Admin UI and when I pull,
amplify pull --appId d4kj5li5xk9a1 --envName dev, I am getting the following error:
Could not initialize 'dev': ENOENT: no such file or directory, realpath 'D:\Developer\Vue_Applications\shortfall\amplify.temp#current-cloud-backend\a
pi\shortfall\build\states\initial-stack\states\initial-stack\states\initial-stack\states\initial-stack\states\initial-stack\states\initial-stack\states\initial-stack\states\initial-stack'

Hi @MachariaK
Can you tell me the exact repro steps so that I can reproduce this error?
Also, which OS platform are you using?

I can't replicate this but I am still facing the same problem. I have a lot of suggestions but none is working.

I changed a model in the backend and this must have caused the error. I have however migrated the project and it's working fine now.

@MachariaK how did you managed to copy data from one model of an environment to a model of another enverionment? did you do this?


libconnman-qt5: Don't harcode libdir to fix multilib build

Signed-off-by: Andreas Oberritter<EMAIL_ADDRESS>

Pushed to master and thud, thanks!
9fc65c0..201fcf2  master -> master
010a29e..201fcf2  thud -> thud


ci: introduce markdown lint

What is the purpose of the change
Introduce markdown lint to check the doc format.
Reference: https://github.com/DavidAnson/markdownlint/tree/v0.32.1/doc

MD001/heading-increment : Heading levels should only increment by one level at a time
MD003/heading-style : Heading style
MD004/ul-style : Unordered list style
MD005/list-indent : Inconsistent indentation for list items at the same level
MD007/ul-indent : Unordered list indentation
MD009/no-trailing-spaces : No trailing spaces
MD010/no-hard-tabs : Hard tab
MD011/no-reversed-links : Reversed link syntax
MD012/no-multiple-blanks : Multiple consecutive blank lines
MD022/blanks-around-headings : Headings should be surrounded by blank lines
MD023/heading-start-left : Headings must start at the beginning of the line
MD030/list-marker-space : Spaces after list markers
MD031/blanks-around-fences : Fenced code blocks should be surrounded by blank lines
MD032/blanks-around-lists : Lists should be surrounded by blank lines
MD034/no-bare-urls : Bare URL used
MD042/no-empty-links : No empty links
MD046/code-block-style : Code block style
MD047/single-trailing-newline : Files should end with a single newline character
MD048/code-fence-style : Code fence style
MD049/emphasis-style : Emphasis style
MD050/strong-style : Strong style

Brief changelog
Introduce markdown lint to check the doc format.
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue filed for the change(usually before you start working on it).  Trivial changes such as typos do not necessitate a GITHUB issue. Your pull request should solely focus on addressing this specific issue without incorporating any other modifications. Each PR should resolve a single issue.
[ ] Format the pull request title like doc: add an operations guide document. Each commit within the pull request should contain a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Test your code locally by running npm run build, and make sure it works as expected.
[ ] Ensure that no files under the build folder are included in the commit, as well as the package-lock.json and yarn.lock files.

另外，需要考虑开发者如何在本地执行markdown lint修复

@liuqiufeng Currently, lint rules only work on README.md as a demo in this PR. We can easily get the lint check result:
$ markdownlint '**/*.md' --ignore src \
             --ignore node_modules \
             --ignore blog \
             --ignore i18n \
             --ignore versioned_docs \
             --ignore versioned_sidebars \
             --ignore docs \
             --ignore readme

Output:
README.md:5:25 MD034/no-bare-urls Bare URL used [Context: "https://seata.io"]
README.md:7 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "# README.md"]
README.md:7 MD025/single-title/single-h1 Multiple top-level headings in the same document [Context: "# README.md"]
README.md:8 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- en [English](README.md)"]
README.md:11 MD001/heading-increment Heading levels should only increment by one level at a time [Expected: h2; Actual: h3]
README.md:19 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "3. Visit"]
README.md:20 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:20 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:30 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "3. Visit"]
README.md:31 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:31 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:35 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Requirements"]
README.md:39 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Note for i18n"]
README.md:42 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Overview of the Website Structure"]
README.md:46 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:46 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:88 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Directories Related to Adding Documents"]
README.md:105 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Summary of Adding Documents"]
README.md:120:1 MD027/no-multiple-space-blockquote Multiple spaces after blockquote symbol [Context: ">  Notes: Corresponding to Chi..."]
README.md:122 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]
README.md:123 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## How To Add Blogs"]
README.md:126 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Directories Related to Adding Blogs"]
README.md:134 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Summary of Adding Blogs"]
README.md:141:1 MD027/no-multiple-space-blockquote Multiple spaces after blockquote symbol [Context: ">  Notes: Corresponding to Chi..."]
README.md:146 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:146 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:155 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:157:31 MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]
README.md:157 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "1. `title` can not include `:`"]
README.md:161 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Note for .md file"]
README.md:162 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "1. Do not use un correct html ..."]
README.md:167:1 MD004/ul-style Unordered list style [Expected: dash; Actual: plus]

Obviously we just follow the output info and fix them one by one.
However, there are too many existed docs without beening checked by lint in other directories.
Then IMHO, we can ingore them temporarily in this PR, and fix them in new several PRs.

@liuqiufeng Currently, lint rules only work on README.md as a demo in this PR. We can easily get the lint check result:
$ markdownlint '**/*.md' --ignore src \
             --ignore node_modules \
             --ignore blog \
             --ignore i18n \
             --ignore versioned_docs \
             --ignore versioned_sidebars \
             --ignore docs \
             --ignore readme

Output:
README.md:5:25 MD034/no-bare-urls Bare URL used [Context: "https://seata.io"]
README.md:7 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "# README.md"]
README.md:7 MD025/single-title/single-h1 Multiple top-level headings in the same document [Context: "# README.md"]
README.md:8 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- en [English](README.md)"]
README.md:11 MD001/heading-increment Heading levels should only increment by one level at a time [Expected: h2; Actual: h3]
README.md:19 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "3. Visit"]
README.md:20 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:20 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:30 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "3. Visit"]
README.md:31 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:31 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:35 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Requirements"]
README.md:39 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Note for i18n"]
README.md:42 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Overview of the Website Structure"]
README.md:46 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:46 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:88 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Directories Related to Adding Documents"]
README.md:105 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Summary of Adding Documents"]
README.md:120:1 MD027/no-multiple-space-blockquote Multiple spaces after blockquote symbol [Context: ">  Notes: Corresponding to Chi..."]
README.md:122 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]
README.md:123 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## How To Add Blogs"]
README.md:126 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Directories Related to Adding Blogs"]
README.md:134 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### Summary of Adding Blogs"]
README.md:141:1 MD027/no-multiple-space-blockquote Multiple spaces after blockquote symbol [Context: ">  Notes: Corresponding to Chi..."]
README.md:146 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:146 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:155 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]
README.md:157:31 MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]
README.md:157 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "1. `title` can not include `:`"]
README.md:161 MD022/blanks-around-headings Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## Note for .md file"]
README.md:162 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "1. Do not use un correct html ..."]
README.md:167:1 MD004/ul-style Unordered list style [Expected: dash; Actual: plus]

Obviously we just follow the output info and fix them one by one.
However, there are too many existed docs without beening checked by lint in other directories.
Then IMHO, we can ingore them temporarily in this PR, and fix them in new several PRs.

目前的写法需要提交pr之后运行ci才能知道哪里写的有问题，或使用vscode的markdownlint插件，或者本地全局安装markdownlint。不太方便。
The current writeup needs to be submitted to pr and then run ci to see where the writeup is wrong,or using the markdownlint plugin for vscode,or local global installation of markdownlint package.not very convenient.
我的意思是，增加一个开发依赖markdownlint，再加一个npm命令，只需要跑一下就可以自动扫描或自动修复。
I mean, add a development dependency markdownlint and an npm command to just run it to autoscan or autofix.

I mean, add a development dependency markdownlint and an npm command to just run it to autoscan or autofix.

@liuqiufeng Done. Now we can use npm run lint to conveniently check the doc style : )

Need workflow approval @liuqiufeng @slievrly broken ci fixed.

将seata.io都改为seata.apache.org Replace all seata.io with seata.apache.org

@funky-eyes done.


Odd culture conversion for CK2 exports

Several cultures that have never existed in EU4 are predominant in several regions in Victoria, for example the majority of Anatolia is Turkish while the Turkish culture has never touched Anatolia in neither EU4 nor CK2. Another example is English/Irish/Welsh cultures being the majority in regions that have been Norse since the 800s lore-wise. The "original culture" of these provinces in CK2's exported mod is Norse, which is why I believe this issue is caused by the converter using the vanilla "original culture"s of provinces to determine culture ratios in pops, instead of the ones in the mod.

It has fixed the issue for the most part, however there still are some provinces that have the default original culture and religions used for them. I'm currently checking to see what's different in the EU4 files of these specific provinces.

Ah, oddly the pockets that aren't correctly converted are the exact pockets that were correctly converted before.

I think I've found the issue. The EU4 default culture and religion load first, and the ones from them mod second, in the savegame for the provinces that are correctly converted.
But the exact opposite is happening in the ones that are converted incorrectly.

Drat, I had assumed a consistent order. Back to the drawing board.

Okay, I've tried another fix that I think should sort it. Please try that out.

The issue still persists, these provinces are treated as if they were converted in 1444.

I had the logic backwards, doh! Mind testing again?

Yep, the issue's gone!

Oh thank goodness, the next thing to try would have been a ton of effort.


⚠ ClamAV has degraded performance

In e20636e, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 201 ms

Resolved: ClamAV performance has improved in 5541dbe.


Fix: Facility page view in mobile phones ..

Proposed Changes:

Fixes #8541
Resolved card stretching issue on the Facility tab for iPhone 15 Pro Max devices.
Ensured proper responsiveness of the card layout across all pages.
Adjusted CSS for the first-page layout to prevent content overflow on larger screen sizes.

@ohcnetwork/care-fe-code-reviewers
Merge Checklist:

[ ] Add specs that demonstrate bug / test a new feature.
[ ] Update product documentation.
[ ] Ensure that UI text is kept in I18n files.
[ ] Prep screenshot or demo video for changelog entry, and attach it to issue.
[ ] Request for Peer Reviews.
[ ] Completion of QA.

LGTM