ModuleNotFoundError: No module named 'jax.experimental.maps' after main-brunch update (#662)

Hi - after yesterday's code update I am getting the following error. Any advise? I am using the Jax-levanter docker image
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/levanter/src/levanter/main/train_lm.py", line 220, in <module>
    levanter.config.main(main)()
  File "/levanter/src/levanter/config.py", line 84, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/levanter/src/levanter/main/train_lm.py", line 119, in main
    Vocab = round_axis_for_partitioning(Axis("vocab", vocab_size), parameter_axis_mapping)
  File "/opt/haliax/src/haliax/partitioning.py", line 597, in round_axis_for_partitioning
    size = physical_axis_size(axis, mapping)
  File "/opt/haliax/src/haliax/partitioning.py", line 566, in physical_axis_size
    mesh = _get_mesh()
  File "/opt/haliax/src/haliax/partitioning.py", line 606, in _get_mesh
    from jax.experimental.maps import thread_resources
ModuleNotFoundError: No module named 'jax.experimental.maps'

EDIT: Actually I now see the main error with previous repo HEAD, which is weird cause yesterday the container was working fine. You think something has change on the NVIDA-image ?

yeah looks like they moved/removed jax.experimental.maps. the container sticks close to JAX head, which we don't.

Once I merge https://github.com/stanford-crfm/haliax/pull/102 and a suitable interval for the package to propagate, you should be able to update to the latest dev version of haliax (probably 308 or 309)

try pip install haliax==1.4.dev310 and see if it fixes

Thanks for your response.
This caused a series of errors  that is hard for me trace exactly so I copy the full output for now:
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/levanter/src/levanter/main/train_lm.py", line 220, in
levanter.config.main(main)()
File "/levanter/src/levanter/config.py", line 84, in wrapper_inner
response = fn(cfg, *args, **kwargs)
File "/levanter/src/levanter/main/train_lm.py", line 215, in main
trainer.train(state, train_loader)
File "/levanter/src/levanter/trainer.py", line 403, in train
for info in self.training_steps(state, train_loader, run_hooks=run_hooks):
File "/levanter/src/levanter/trainer.py", line 386, in training_steps
info = self.train_step(state, example)
File "/levanter/src/levanter/trainer.py", line 370, in train_step
loss, new_state = self._jit_train_step_fn(state, *batch, **batch_kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 261, in call
return self._call(False, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/equinox/_module.py", line 1053, in call
return self.func(self.self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 315, in _call
output_shape = _cached_filter_eval_shape(self._fn, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 546, in _cached_filter_eval_shape
_eval_shape_cache[static] = eqx.filter_eval_shape(fun, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/equinox/_eval_shape.py", line 38, in filter_eval_shape
dynamic_out, static_out = jax.eval_shape(ft.partial(_fn, static), dynamic)
File "/levanter/src/levanter/trainer.py", line 498, in _train_step
loss, grads = self._compute_gradients_microbatched(self.loss_fn, model, *batch, **batch_kwargs, key=key)
File "/levanter/src/levanter/trainer.py", line 515, in _compute_gradients_microbatched
return grad_fn(model, *batch, **batch_kwargs)
File "/levanter/src/levanter/grad_accum.py", line 92, in wrapped_fn
r_shape = eqx.filter_eval_shape(fn, *args, **kwargs)
File "/levanter/src/levanter/trainer.py", line 191, in fn
return _ensure_scalar(self._raw_loss_function(model, *batch, **batch_kwargs))
File "/levanter/src/levanter/types.py", line 75, in call
return model.compute_loss(*inputs, reduction=reduction, reduction_axis=reduction_axis, **kwargs)
File "/levanter/src/levanter/models/lm_model.py", line 129, in compute_loss
logits = self(example.tokens, example.attn_mask, key=key)
File "/levanter/src/levanter/models/gpt2.py", line 399, in call
x = self.transformer(x, attn_mask, key=k_transformer)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/levanter/src/levanter/models/gpt2.py", line 298, in call
x = self.blocks.fold(x, attn_mask, hax.arange(self.config.Layers), key=keys)
File "/usr/local/lib/python3.10/dist-packages/haliax/nn/scan.py", line 221, in fold
return haliax.fold(do_block, self.Block)(init, self.stacked, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 202, in scanned_f
return scan_preconfig(init, *args, **kwargs)[0]
File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 134, in scanned_f
carry, ys = lax.scan(wrapped_fn, init, leaves, reverse=reverse, unroll=unroll)
File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 127, in wrapped_fn
carry, y = f(carry, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 197, in scan_compatible_fn
return fn(carry, args, **kwargs), None
File "/usr/local/lib/python3.10/dist-packages/haliax/jax_utils.py", line 75, in wrapper
dynamic_out, static_out = checkpointed_fun(static, dynamic)
File "/usr/local/lib/python3.10/dist-packages/haliax/jax_utils.py", line 66, in _fn
_out = fun(_args, **_kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/nn/scan.py", line 225, in _do_block
return block(carry, *extra_args, **extra_kwargs)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/levanter/src/levanter/models/gpt2.py", line 268, in call
attn_output = self.attn(self.ln_1(x), mask=mask, layer_idx=layer_idx, key=k1)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/levanter/src/levanter/models/gpt2.py", line 200, in call
attn_output = dot_product_attention(
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/levanter/src/levanter/models/attention.py", line 119, in dot_product_attention
attention_out = _try_te_attention(
File "/levanter/src/levanter/models/attention.py", line 242, in _try_te_attention
return _te_flash_attention(
File "/levanter/src/levanter/models/attention.py", line 313, in _te_flash_attention
from transformer_engine.jax.fused_attn import fused_attn  # noqa: F401
File "/usr/local/lib/python3.10/dist-packages/transformer_engine/init.py", line 10, in
import transformer_engine.common
File "/usr/local/lib/python3.10/dist-packages/transformer_engine/common/init.py", line 107, in
_TE_LIB_CTYPES = _load_library()
File "/usr/local/lib/python3.10/dist-packages/transformer_engine/common/init.py", line 78, in _load_library
return ctypes.CDLL(so_path, mode=ctypes.RTLD_GLOBAL)
File "/usr/lib/python3.10/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/python3.10/dist-packages/transformer_engine/libtransformer_engine.so: undefined symbol: cudnnGetLastErrorString
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/levanter/src/levanter/main/train_lm.py", line 220, in
levanter.config.main(main)()
File "/levanter/src/levanter/config.py", line 84, in wrapper_inner
response = fn(cfg, *args, **kwargs)
File "/levanter/src/levanter/main/train_lm.py", line 215, in main
trainer.train(state, train_loader)
File "/levanter/src/levanter/trainer.py", line 403, in train
for info in self.training_steps(state, train_loader, run_hooks=run_hooks):
File "/levanter/src/levanter/trainer.py", line 386, in training_steps
info = self.train_step(state, example)
File "/levanter/src/levanter/trainer.py", line 370, in train_step
loss, new_state = self._jit_train_step_fn(state, *batch, **batch_kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 261, in call
return self._call(False, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/equinox/_module.py", line 1053, in call
return self.func(self.self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 315, in _call
output_shape = _cached_filter_eval_shape(self._fn, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 546, in _cached_filter_eval_shape
_eval_shape_cache[static] = eqx.filter_eval_shape(fun, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/equinox/_eval_shape.py", line 38, in filter_eval_shape
dynamic_out, static_out = jax.eval_shape(ft.partial(_fn, static), dynamic)
File "/levanter/src/levanter/trainer.py", line 498, in _train_step
loss, grads = self._compute_gradients_microbatched(self.loss_fn, model, *batch, **batch_kwargs, key=key)
File "/levanter/src/levanter/trainer.py", line 515, in _compute_gradients_microbatched
return grad_fn(model, *batch, **batch_kwargs)
File "/levanter/src/levanter/grad_accum.py", line 92, in wrapped_fn
r_shape = eqx.filter_eval_shape(fn, *args, **kwargs)
File "/levanter/src/levanter/trainer.py", line 191, in fn
return _ensure_scalar(self._raw_loss_function(model, *batch, **batch_kwargs))
File "/levanter/src/levanter/types.py", line 75, in call
return model.compute_loss(*inputs, reduction=reduction, reduction_axis=reduction_axis, **kwargs)
File "/levanter/src/levanter/models/lm_model.py", line 129, in compute_loss
logits = self(example.tokens, example.attn_mask, key=key)
File "/levanter/src/levanter/models/gpt2.py", line 399, in call
x = self.transformer(x, attn_mask, key=k_transformer)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/levanter/src/levanter/models/gpt2.py", line 298, in call
x = self.blocks.fold(x, attn_mask, hax.arange(self.config.Layers), key=keys)
File "/usr/local/lib/python3.10/dist-packages/haliax/nn/scan.py", line 221, in fold
return haliax.fold(do_block, self.Block)(init, self.stacked, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 202, in scanned_f
return scan_preconfig(init, *args, **kwargs)[0]
File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 134, in scanned_f
carry, ys = lax.scan(wrapped_fn, init, leaves, reverse=reverse, unroll=unroll)
File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 127, in wrapped_fn
carry, y = f(carry, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 197, in scan_compatible_fn
return fn(carry, args, **kwargs), None
File "/usr/local/lib/python3.10/dist-packages/haliax/jax_utils.py", line 75, in wrapper
dynamic_out, static_out = checkpointed_fun(static, dynamic)
File "/usr/local/lib/python3.10/dist-packages/haliax/jax_utils.py", line 66, in _fn
_out = fun(_args, **_kwargs)
File "/usr/local/lib/python3.10/dist-packages/haliax/nn/scan.py", line 225, in _do_block
return block(carry, *extra_args, **extra_kwargs)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/levanter/src/levanter/models/gpt2.py", line 268, in call
attn_output = self.attn(self.ln_1(x), mask=mask, layer_idx=layer_idx, key=k1)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/levanter/src/levanter/models/gpt2.py", line 200, in call
attn_output = dot_product_attention(
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/levanter/src/levanter/models/attention.py", line 119, in dot_product_attention
attention_out = _try_te_attention(
File "/levanter/src/levanter/models/attention.py", line 242, in _try_te_attention
return _te_flash_attention(
File "/levanter/src/levanter/models/attention.py", line 313, in _te_flash_attention
from transformer_engine.jax.fused_attn import fused_attn  # noqa: F401
File "/usr/local/lib/python3.10/dist-packages/transformer_engine/init.py", line 10, in
import transformer_engine.common
File "/usr/local/lib/python3.10/dist-packages/transformer_engine/common/init.py", line 107, in
_TE_LIB_CTYPES = _load_library()
File "/usr/local/lib/python3.10/dist-packages/transformer_engine/common/init.py", line 78, in _load_library
return ctypes.CDLL(so_path, mode=ctypes.RTLD_GLOBAL)
File "/usr/lib/python3.10/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/python3.10/dist-packages/transformer_engine/libtransformer_engine.so: undefined symbol: cudnnGetLastErrorString
2024-07-25 20:44:28,532 WARNING worker.py:1450 -- SIGTERM handler is not set because current thread is not the main thread.
/usr/lib/python3.10/subprocess.py:1796: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
self.pid = _posixsubprocess.fork_exec(
wandb:
wandb: Run history:
wandb:              preprocessing//train/chunks ▁
wandb:            preprocessing//train/finished ▁
wandb:           preprocessing//train/input_ids ▁
wandb:                preprocessing//train/rows ▁
wandb:              preprocessing//train/shards ▁
wandb:      preprocessing//train/token_type_ids ▁
wandb:         preprocessing//validation/chunks ▁
wandb:       preprocessing//validation/finished ▁
wandb:      preprocessing//validation/input_ids ▁
wandb:           preprocessing//validation/rows ▁
wandb:         preprocessing//validation/shards ▁
wandb: preprocessing//validation/token_type_ids ▁
wandb:
wandb: Run summary:
wandb:                                  backend gpu
wandb:                              num_devices 1
wandb:                                num_hosts 1
wandb:                          parameter_count 359708672
wandb:              preprocessing//train/chunks 1
wandb:            preprocessing//train/finished 1
wandb:           preprocessing//train/input_ids 285696
wandb:                preprocessing//train/rows 279
wandb:              preprocessing//train/shards 1
wandb:      preprocessing//train/token_type_ids 285696
wandb:         preprocessing//validation/chunks 1
wandb:       preprocessing//validation/finished 1
wandb:      preprocessing//validation/input_ids 28672
wandb:           preprocessing//validation/rows 28
wandb:         preprocessing//validation/shards 1
wandb: preprocessing//validation/token_type_ids 28672
wandb:                   throughput/device_kind NVIDIA A10G
wandb:             throughput/flops_per_example<PHONE_NUMBER>608.0
wandb:             throughput/theoretical_flops<PHONE_NUMBER>00000.0
wandb:  throughput/theoretical_flops_per_device<PHONE_NUMBER>00000.0
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /levanter/wandb/offline-run-20240725_204346-cirlyl1x
wandb: Find logs at: ./wandb/offline-run-20240725_204346-cirlyl1x/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with wandb.require("core")! See https://wandb.me/wandb-core for more information.
2024-07-25 20:44:30,777 INFO worker.py:1779 -- Started a local Ray instance. View the dashboard at <IP_ADDRESS>:8265
2024-07-25T20:44:31 - 0 - ShardCache.cache/train - shard_cache.py:1418 - ERROR :: Error while reading from shard cache.
Traceback (most recent call last):
File "/levanter/src/levanter/data/shard_cache.py", line 1406, in iter_batches_from_chunks
chunk = self._get_chunk_unmapped(i)
File "/levanter/src/levanter/data/shard_cache.py", line 1336, in _get_chunk_unmapped
chunk = ray.get(self._broker.get_chunk.remote(mapped_index), timeout=current_timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/actor.py", line 202, in remote
return self._remote(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/util/tracing/tracing_helper.py", line 426, in _start_span
return method(self, args, kwargs, *_args, **_kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/actor.py", line 330, in _remote
return invocation(args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/actor.py", line 311, in invocation
return actor._actor_method_call(
File "/usr/local/lib/python3.10/dist-packages/ray/actor.py", line 1460, in _actor_method_call
object_refs = worker.core_worker.submit_actor_task(
File "python/ray/_raylet.pyx", line 4258, in ray._raylet.CoreWorker.submit_actor_task
File "python/ray/_raylet.pyx", line 4313, in ray._raylet.CoreWorker.submit_actor_task
Exception: Failed to submit task to actor ActorID(01757d23543161964c52264701000000) due to b"Can't find actor 01757d23543161964c52264701000000. It might be dead or it's from a different cluster"
root@215bee9de542:/levanter#

Please ignore and let me retest - I was wasn't on Levanter head so that might be it. Will follow up

Yeah same output unfortunately. Is there a way I can access older jax containers?
INFO:levanter.distributed:Not initializing jax.distributed because no distributed config was provided, and no cluster was detected.
INFO:jax._src.xla_bridge:Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA
INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
INFO:levanter.trainer:Setting run id to oz48brcz
2024-07-25T21:34:46 - 0 - levanter.tracker.wandb - wandb.py:200 - INFO :: Setting wandb code_dir to /levanter
2024-07-25T21:34:46 - 0 - wandb.sdk.lib.gitlib - gitlib.py:92 - ERROR :: git root error: Cmd('git') failed due to: exit code(128)
  cmdline: git rev-parse --show-toplevel
  stderr: 'fatal: detected dubious ownership in repository at '/levanter'
To add an exception for this directory, call:

        git config --global --add safe.directory /levanter'
2024-07-25T21:34:46 - 0 - wandb.sdk.lib.gitlib - gitlib.py:92 - ERROR :: git root error: Cmd('git') failed due to: exit code(128)
  cmdline: git rev-parse --show-toplevel
  stderr: 'fatal: detected dubious ownership in repository at '/levanter'
To add an exception for this directory, call:

        git config --global --add safe.directory /levanter'
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose "Don't visualize my results"
wandb: WARNING `resume` will be ignored since W&B syncing is set to `offline`. Starting a new run with run id oz48brcz.
wandb: Tracking run with wandb version 0.17.5
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
2024-07-25T21:34:49 - 0 - levanter.distributed - distributed.py:215 - INFO :: No auto-discovered ray address found. Using ray.init('local').
2024-07-25T21:34:49 - 0 - levanter.distributed - distributed.py:267 - INFO :: ray.init(address='local', namespace='levanter', **{})
/usr/lib/python3.10/subprocess.py:1796: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
  self.pid = _posixsubprocess.fork_exec(
/usr/lib/python3.10/subprocess.py:1796: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
  self.pid = _posixsubprocess.fork_exec(
2024-07-25 21:34:51,416 INFO worker.py:1779 -- Started a local Ray instance. View the dashboard at <IP_ADDRESS>:8265
2024-07-25T21:34:52 - 0 - levanter.tracker.wandb - wandb.py:200 - INFO :: Setting wandb code_dir to /levanter
2024-07-25T21:34:52 - 0 - levanter.tracker.wandb - wandb.py:200 - INFO :: Setting wandb code_dir to /levanter
train:   0%|                                                                                | 0/50 [00:00<?, ?it/s]2024-07-25T21:34:53 - 0 - levanter.data.shard_cache - shard_cache.py:1266 - INFO :: Loading cache from /cache/validation
2024-07-25T21:34:53 - 0 - levanter.data.text - text.py:692 - INFO :: Building cache for validation...
2024-07-25T21:34:53 - 0 - levanter.data.shard_cache - shard_cache.py:1266 - INFO :: Loading cache from /cache/validation
(ChunkCacheBuilder pid=505) 2024-07-25 21:34:59,609 - levanter.data.shard_cache.builder::cache/validation - INFO - Starting cache build for 1 shards
2024-07-25T21:35:06 - 0 - levanter.data.text - text.py:256 - INFO :: Cache /cache/validation is complete.
2024-07-25T21:35:06 - 0 - levanter.data.shard_cache - shard_cache.py:1266 - INFO :: Loading cache from /cache/train
2024-07-25T21:35:06 - 0 - levanter.data.text - text.py:692 - INFO :: Building cache for train...
2024-07-25T21:35:06 - 0 - levanter.data.shard_cache - shard_cache.py:1266 - INFO :: Loading cache from /cache/train
2024-07-25T21:35:06 - 0 - preprocessing..validation - metrics_monitor.py:143 - INFO ::  done: Shards: 0 | Chunks: 1 | Docs: 28
2024-07-25T21:35:06 - 0 - preprocessing..validation - metrics_monitor.py:143 - INFO ::  done: Shards: 1 | Chunks: 1 | Docs: 28
2024-07-25T21:35:06 - 0 - preprocessing..validation - metrics_monitor.py:150 - INFO :: Cache creation finished
(ChunkCacheBroker pid=460) 2024-07-25 21:35:06,723 - levanter.data.shard_cache - INFO - Finalizing cache /cache/validation...
(ChunkCacheBuilder pid=505) 2024-07-25 21:35:06,718 - levanter.data.shard_cache - INFO - Shard valid_txt finished
2024-07-25T21:35:10 - 0 - levanter.data.text - text.py:258 - INFO :: Cache /cache/train is incomplete. This will block until at least one chunk per process is complete.
(ChunkCacheBuilder pid=699) 2024-07-25 21:35:13,905 - levanter.data.shard_cache.builder::cache/train - INFO - Starting cache build for 1 shards
2024-07-25T21:35:14 - 0 - __main__ - train_lm.py:129 - INFO :: No training checkpoint found. Initializing model from HF checkpoint 'stanford-crfm/music-medium-800k'
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.96k/1.96k [00:00<00:00, 8.35MB/s]
config.json:   0%|                                                                                | 0.00/1.96k [00:00<?, ?B/s2024-07-25T21:35:21 - 0 - preprocessing..train - metrics_monitor.py:143 - INFO ::  done: Shards: 0 | Chunks: 1 | Docs: 279                                          | 262M/1.44G [00:05<00:23, 51.0MB/s]
2024-07-25T21:35:21 - 0 - preprocessing..train - metrics_monitor.py:143 - INFO ::  done: Shards: 1 | Chunks: 1 | Docs: 279
2024-07-25T21:35:21 - 0 - preprocessing..train - metrics_monitor.py:143 - INFO ::  done: Shards: 1 | Chunks: 1 | Docs: 279
2024-07-25T21:35:21 - 0 - preprocessing..train - metrics_monitor.py:150 - INFO :: Cache creation finished
(ChunkCacheBroker pid=664) 2024-07-25 21:35:21,188 - levanter.data.shard_cache - INFO - Finalizing cache /cache/train...
(ChunkCacheBuilder pid=699) 2024-07-25 21:35:21,183 - levanter.data.shard_cache - INFO - Shard train_txt finished
pytorch_model.bin: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.44G/1.44G [00:38<00:00, 37.6MB/s]
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 293/293 [00:01<00:00, 189.18it/s]
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.<00:00, 182.51it/s]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/levanter/src/levanter/main/train_lm.py", line 220, in <module>
    levanter.config.main(main)()
  File "/levanter/src/levanter/config.py", line 84, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/levanter/src/levanter/main/train_lm.py", line 215, in main
    trainer.train(state, train_loader)
  File "/levanter/src/levanter/trainer.py", line 403, in train
    for info in self.training_steps(state, train_loader, run_hooks=run_hooks):
  File "/levanter/src/levanter/trainer.py", line 386, in training_steps
    info = self.train_step(state, example)
  File "/levanter/src/levanter/trainer.py", line 370, in train_step
    loss, new_state = self._jit_train_step_fn(state, *batch, **batch_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 261, in __call__
    return self._call(False, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/equinox/_module.py", line 1053, in __call__
    return self.__func__(self.__self__, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 315, in _call
    output_shape = _cached_filter_eval_shape(self._fn, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 546, in _cached_filter_eval_shape
    _eval_shape_cache[static] = eqx.filter_eval_shape(fun, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/equinox/_eval_shape.py", line 38, in filter_eval_shape
    dynamic_out, static_out = jax.eval_shape(ft.partial(_fn, static), dynamic)
  File "/levanter/src/levanter/trainer.py", line 498, in _train_step
    loss, grads = self._compute_gradients_microbatched(self.loss_fn, model, *batch, **batch_kwargs, key=key)
  File "/levanter/src/levanter/trainer.py", line 515, in _compute_gradients_microbatched
    return grad_fn(model, *batch, **batch_kwargs)
  File "/levanter/src/levanter/grad_accum.py", line 92, in wrapped_fn
    r_shape = eqx.filter_eval_shape(fn, *args, **kwargs)
  File "/levanter/src/levanter/trainer.py", line 191, in fn
    return _ensure_scalar(self._raw_loss_function(model, *batch, **batch_kwargs))
  File "/levanter/src/levanter/types.py", line 75, in __call__
    return model.compute_loss(*inputs, reduction=reduction, reduction_axis=reduction_axis, **kwargs)
  File "/levanter/src/levanter/models/lm_model.py", line 129, in compute_loss
    logits = self(example.tokens, example.attn_mask, key=key)
  File "/levanter/src/levanter/models/gpt2.py", line 399, in __call__
    x = self.transformer(x, attn_mask, key=k_transformer)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/levanter/src/levanter/models/gpt2.py", line 298, in __call__
    x = self.blocks.fold(x, attn_mask, hax.arange(self.config.Layers), key=keys)
  File "/usr/local/lib/python3.10/dist-packages/haliax/nn/scan.py", line 221, in fold
    return haliax.fold(do_block, self.Block)(init, self.stacked, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 202, in scanned_f
    return scan_preconfig(init, *args, **kwargs)[0]
  File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 134, in scanned_f
    carry, ys = lax.scan(wrapped_fn, init, leaves, reverse=reverse, unroll=unroll)
  File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 127, in wrapped_fn
    carry, y = f(carry, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 197, in scan_compatible_fn
    return fn(carry, *args, **kwargs), None
  File "/usr/local/lib/python3.10/dist-packages/haliax/jax_utils.py", line 75, in wrapper
    dynamic_out, static_out = checkpointed_fun(static, dynamic)
  File "/usr/local/lib/python3.10/dist-packages/haliax/jax_utils.py", line 66, in _fn
    _out = fun(*_args, **_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/nn/scan.py", line 225, in _do_block
    return block(carry, *extra_args, **extra_kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/levanter/src/levanter/models/gpt2.py", line 268, in __call__
    attn_output = self.attn(self.ln_1(x), mask=mask, layer_idx=layer_idx, key=k1)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/levanter/src/levanter/models/gpt2.py", line 200, in __call__
    attn_output = dot_product_attention(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/levanter/src/levanter/models/attention.py", line 119, in dot_product_attention
    attention_out = _try_te_attention(
  File "/levanter/src/levanter/models/attention.py", line 242, in _try_te_attention
    return _te_flash_attention(
  File "/levanter/src/levanter/models/attention.py", line 313, in _te_flash_attention
    from transformer_engine.jax.fused_attn import fused_attn  # noqa: F401
  File "/usr/local/lib/python3.10/dist-packages/transformer_engine/__init__.py", line 10, in <module>
    import transformer_engine.common
  File "/usr/local/lib/python3.10/dist-packages/transformer_engine/common/__init__.py", line 107, in <module>
    _TE_LIB_CTYPES = _load_library()
  File "/usr/local/lib/python3.10/dist-packages/transformer_engine/common/__init__.py", line 78, in _load_library
    return ctypes.CDLL(so_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/python3.10/dist-packages/transformer_engine/libtransformer_engine.so: undefined symbol: cudnnGetLastErrorString
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/levanter/src/levanter/main/train_lm.py", line 220, in <module>
    levanter.config.main(main)()
  File "/levanter/src/levanter/config.py", line 84, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/levanter/src/levanter/main/train_lm.py", line 215, in main
    trainer.train(state, train_loader)
  File "/levanter/src/levanter/trainer.py", line 403, in train
    for info in self.training_steps(state, train_loader, run_hooks=run_hooks):
  File "/levanter/src/levanter/trainer.py", line 386, in training_steps
    info = self.train_step(state, example)
  File "/levanter/src/levanter/trainer.py", line 370, in train_step
    loss, new_state = self._jit_train_step_fn(state, *batch, **batch_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 261, in __call__
    return self._call(False, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/equinox/_module.py", line 1053, in __call__
    return self.__func__(self.__self__, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 315, in _call
    output_shape = _cached_filter_eval_shape(self._fn, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/partitioning.py", line 546, in _cached_filter_eval_shape
    _eval_shape_cache[static] = eqx.filter_eval_shape(fun, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/equinox/_eval_shape.py", line 38, in filter_eval_shape
    dynamic_out, static_out = jax.eval_shape(ft.partial(_fn, static), dynamic)
  File "/levanter/src/levanter/trainer.py", line 498, in _train_step
    loss, grads = self._compute_gradients_microbatched(self.loss_fn, model, *batch, **batch_kwargs, key=key)
  File "/levanter/src/levanter/trainer.py", line 515, in _compute_gradients_microbatched
    return grad_fn(model, *batch, **batch_kwargs)
  File "/levanter/src/levanter/grad_accum.py", line 92, in wrapped_fn
    r_shape = eqx.filter_eval_shape(fn, *args, **kwargs)
  File "/levanter/src/levanter/trainer.py", line 191, in fn
    return _ensure_scalar(self._raw_loss_function(model, *batch, **batch_kwargs))
  File "/levanter/src/levanter/types.py", line 75, in __call__
    return model.compute_loss(*inputs, reduction=reduction, reduction_axis=reduction_axis, **kwargs)
  File "/levanter/src/levanter/models/lm_model.py", line 129, in compute_loss
    logits = self(example.tokens, example.attn_mask, key=key)
  File "/levanter/src/levanter/models/gpt2.py", line 399, in __call__
    x = self.transformer(x, attn_mask, key=k_transformer)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/levanter/src/levanter/models/gpt2.py", line 298, in __call__
    x = self.blocks.fold(x, attn_mask, hax.arange(self.config.Layers), key=keys)
  File "/usr/local/lib/python3.10/dist-packages/haliax/nn/scan.py", line 221, in fold
    return haliax.fold(do_block, self.Block)(init, self.stacked, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 202, in scanned_f
    return scan_preconfig(init, *args, **kwargs)[0]
  File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 134, in scanned_f
    carry, ys = lax.scan(wrapped_fn, init, leaves, reverse=reverse, unroll=unroll)
  File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 127, in wrapped_fn
    carry, y = f(carry, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/hof.py", line 197, in scan_compatible_fn
    return fn(carry, *args, **kwargs), None
  File "/usr/local/lib/python3.10/dist-packages/haliax/jax_utils.py", line 75, in wrapper
    dynamic_out, static_out = checkpointed_fun(static, dynamic)
  File "/usr/local/lib/python3.10/dist-packages/haliax/jax_utils.py", line 66, in _fn
    _out = fun(*_args, **_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/haliax/nn/scan.py", line 225, in _do_block
    return block(carry, *extra_args, **extra_kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/levanter/src/levanter/models/gpt2.py", line 268, in __call__
    attn_output = self.attn(self.ln_1(x), mask=mask, layer_idx=layer_idx, key=k1)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/levanter/src/levanter/models/gpt2.py", line 200, in __call__
    attn_output = dot_product_attention(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/levanter/src/levanter/models/attention.py", line 119, in dot_product_attention
    attention_out = _try_te_attention(
  File "/levanter/src/levanter/models/attention.py", line 242, in _try_te_attention
    return _te_flash_attention(
  File "/levanter/src/levanter/models/attention.py", line 313, in _te_flash_attention
    from transformer_engine.jax.fused_attn import fused_attn  # noqa: F401
  File "/usr/local/lib/python3.10/dist-packages/transformer_engine/__init__.py", line 10, in <module>
    import transformer_engine.common
  File "/usr/local/lib/python3.10/dist-packages/transformer_engine/common/__init__.py", line 107, in <module>
    _TE_LIB_CTYPES = _load_library()
  File "/usr/local/lib/python3.10/dist-packages/transformer_engine/common/__init__.py", line 78, in _load_library
    return ctypes.CDLL(so_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/local/lib/python3.10/dist-packages/transformer_engine/libtransformer_engine.so: undefined symbol: cudnnGetLastErrorString
2024-07-25 21:36:00,899 WARNING worker.py:1450 -- SIGTERM handler is not set because current thread is not the main thread.
/usr/lib/python3.10/subprocess.py:1796: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.
  self.pid = _posixsubprocess.fork_exec(
wandb:
wandb: Run history:
wandb:              preprocessing//train/chunks ▁
wandb:            preprocessing//train/finished ▁
wandb:           preprocessing//train/input_ids ▁
wandb:                preprocessing//train/rows ▁
wandb:              preprocessing//train/shards ▁
wandb:      preprocessing//train/token_type_ids ▁
wandb:         preprocessing//validation/chunks ▁
wandb:       preprocessing//validation/finished ▁
wandb:      preprocessing//validation/input_ids ▁
wandb:           preprocessing//validation/rows ▁
wandb:         preprocessing//validation/shards ▁
wandb: preprocessing//validation/token_type_ids ▁
wandb:
wandb: Run summary:
wandb:                                  backend gpu
wandb:                              num_devices 1
wandb:                                num_hosts 1
wandb:                          parameter_count 359708672
wandb:              preprocessing//train/chunks 1
wandb:            preprocessing//train/finished 1
wandb:           preprocessing//train/input_ids 285696
wandb:                preprocessing//train/rows 279
wandb:              preprocessing//train/shards 1
wandb:      preprocessing//train/token_type_ids 285696
wandb:         preprocessing//validation/chunks 1
wandb:       preprocessing//validation/finished 1
wandb:      preprocessing//validation/input_ids 28672
wandb:           preprocessing//validation/rows 28
wandb:         preprocessing//validation/shards 1
wandb: preprocessing//validation/token_type_ids 28672
wandb:                   throughput/device_kind NVIDIA A10G
wandb:             throughput/flops_per_example<PHONE_NUMBER>608.0
wandb:             throughput/theoretical_flops<PHONE_NUMBER>00000.0
wandb:  throughput/theoretical_flops_per_device<PHONE_NUMBER>00000.0
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /levanter/wandb/offline-run-20240725_213448-oz48brcz
wandb: Find logs at: ./wandb/offline-run-20240725_213448-oz48brcz/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information.
2024-07-25 21:36:03,171 INFO worker.py:1779 -- Started a local Ray instance. View the dashboard at <IP_ADDRESS>:8265
2024-07-25T21:36:04 - 0 - ShardCache.cache/train - shard_cache.py:1418 - ERROR :: Error while reading from shard cache.
Traceback (most recent call last):
  File "/levanter/src/levanter/data/shard_cache.py", line 1406, in iter_batches_from_chunks
    chunk = self._get_chunk_unmapped(i)
  File "/levanter/src/levanter/data/shard_cache.py", line 1336, in _get_chunk_unmapped
    chunk = ray.get(self._broker.get_chunk.remote(mapped_index), timeout=current_timeout)
  File "/usr/local/lib/python3.10/dist-packages/ray/actor.py", line 202, in remote
    return self._remote(args, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/util/tracing/tracing_helper.py", line 426, in _start_span
    return method(self, args, kwargs, *_args, **_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/actor.py", line 330, in _remote
    return invocation(args, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/actor.py", line 311, in invocation
    return actor._actor_method_call(
  File "/usr/local/lib/python3.10/dist-packages/ray/actor.py", line 1460, in _actor_method_call
    object_refs = worker.core_worker.submit_actor_task(
  File "python/ray/_raylet.pyx", line 4258, in ray._raylet.CoreWorker.submit_actor_task
  File "python/ray/_raylet.pyx", line 4313, in ray._raylet.CoreWorker.submit_actor_task
Exception: Failed to submit task to actor ActorID(0e2b26ec3dd9161148784ee601000000) due to b"Can't find actor 0e2b26ec3dd9161148784ee601000000. It might be dead or it's from a different cluster"
Exception in thread ray_print_logs:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 893, in print_logs
    subscriber.subscribe()
  File "python/ray/_raylet.pyx", line 3111, in ray._raylet._GcsSubscriber.subscribe
  File "python/ray/_raylet.pyx", line 586, in ray._raylet.check_status
ray.exceptions.RpcError: recvmsg:Connection reset by peer

this seems like a probelm with the cuda stuff. @DwarKapex Any thoughts?

(You can probably use https://github.com/orgs/nvidia/packages/container/jax/248105500?tag=levanter-2024-07-24 as jax:levanter-2024-07-24)

that did the trick! Thanks so much!

If it is of any help to you here is a slightly more detailed description of what I am trying to do.

I am trying to fine-tune this model which has been built using Levanter
The author of that work suggested in the past the this Levanter code edit  plus installing torch to run the implementation (essentially is a gpt2 model)
With jax:levanter-2024-07-24 this trick seems to work.
With the current JAX image and without this Levanter code edit   I am getting this error
With the current JAX image and with this Levanter code edit   I am getting

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/levanter/src/levanter/main/train_lm.py", line 220, in <module>
    levanter.config.main(main)()
  File "/levanter/src/levanter/config.py", line 84, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/levanter/src/levanter/main/train_lm.py", line 82, in main
    levanter.initialize(config)
  File "/levanter/src/levanter/trainer.py", line 796, in initialize
    trainer_config.initialize()
  File "/levanter/src/levanter/trainer.py", line 627, in initialize
    _initialize_global_tracker(self.tracker, id)
  File "/levanter/src/levanter/trainer.py", line 522, in _initialize_global_tracker
    tracker = config.init(run_id)
  File "/levanter/src/levanter/tracker/wandb.py", line 131, in init
    git_settings = self._git_settings()
  File "/levanter/src/levanter/tracker/wandb.py", line 204, in _git_settings
    sha = self._get_git_sha(code_dir)
  File "/levanter/src/levanter/tracker/wandb.py", line 216, in _get_git_sha
    git_sha = repo.head.commit.hexsha
  File "/usr/local/lib/python3.10/dist-packages/git/refs/symbolic.py", line 297, in _get_commit
    obj = self._get_object()
  File "/usr/local/lib/python3.10/dist-packages/git/refs/symbolic.py", line 288, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "/usr/local/lib/python3.10/dist-packages/git/objects/base.py", line 149, in new_from_sha
    oinfo = repo.odb.info(sha1)
  File "/usr/local/lib/python3.10/dist-packages/git/db.py", line 41, in info
    hexsha, typename, size = self._git.get_object_header(bin_to_hex(binsha))
  File "/usr/local/lib/python3.10/dist-packages/git/cmd.py", line 1678, in get_object_header
    return self.__get_object_header(cmd, ref)
  File "/usr/local/lib/python3.10/dist-packages/git/cmd.py", line 1661, in __get_object_header
    cmd.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

Sharing this in case it helps you guys with debugging. Feel free to follow up if you have any question.

weird. that's easy to workaround but I don't know why it's happening


Maxout over convolutional layer.

I am new to keras/theano. I have been trying to implement maxout for convolutional layer output. I know that there is MaxoutDense layer present in keras and also Convolution2D layer, but I am unable to merge them for my purpose. It might be because I am not able to understand maxout properly.
I am trying to replicate the results of Deep Features for Text Spotting. The maxout network in this paper simply uses max over consecutive filters.
I would appreciate if someone can share their code for maxout for convolutional layers.
With small understanding that I have, I have written small maxout (lambda layer) of my own but it gives me strange error:
img_rows = 24
img_cols = 24
def maxout(x):
out = None
for i in range(0, 32):
Max = K.max(x[:, i * 2:(i + 1) * 2, :, :], axis=1, keepdims=True)
K.expand_dims(Max, dim = 0)
if out == None:
out = Max
else:
out = K.concatenate([out, Max], axis=1)
return out
def maxout_output_shape(input_shape):
shape = list(input_shape)
shape[1] = 16
return tuple(shape)
model = Sequential()
model.add(Convolution2D(32, 3, 3, input_shape=(1, img_rows, img_cols)))
model.add(Lambda(function=maxout, output_shape=maxout_output_shape))
model.compile(optimizer='adam', loss="mse")
X = np.ones((1, 1, 24, 24))
layer_output = K.function([model.layers[0].input],
[model.layers[1].output])
layer_output = layer_output([X])
The errors are:
Using Theano backend.
Traceback (most recent call last):
File "temp.py", line 41, in
layer_output = layer_output([X])
File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.3-py2.7.egg/keras/backend/theano_backend.py", line 518, in call
return self.function(*inputs)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 871, in call
storage_map=getattr(self.fn, 'storage_map', None))
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 859, in call
outputs = self.fn()
ValueError: Input of CAReduce{maximum} has zero-size on axis 1
Apply node that caused the error: Reduce{maximum}{1}(Subtensor{::, int64:int64:}.0)
Toposort index: 51
Inputs types: [TensorType(float32, 4D)]
Inputs shapes: [(1, 0, 22, 22)]
Inputs strides: [(61952, 1936, 88, 4)]
Inputs values: [array([], shape=(1, 0, 22, 22), dtype=float32)]
Outputs clients: [[InplaceDimShuffle{0,x,1,2}(max)]]

@GodOfProbability I think you can close this issue, sunce #3128 solves it :)


receiveValue not called when using resetPassword() or confirmResetPassword()

Describe the bug
I'm trying to implement a "forgot password" flow in my iOS app, and facing an issue when using Amplify.Auth.resetPassword & Amplify.Auth.confirmResetPassword with Combine - I call the functions as instructed in the example provided in the Amplify dev docs, and while the actual action seems to work (for resetPassword - I get an email with a verification code as expected, for confirmResetPassword - the provided new password becomes the one in use - validated using web app and appsync console), I never get an event with a result or error in the sink attached to the Amplify.Publisher.
Steps To Reproduce
Steps to reproduce the behavior:
1. call Amplify.Auth.resetPassword using Amplify.Publisher
2. wait for the request to complete
3. no error or result is triggered in the sink attached to the publisher
4. same goes for Amplify.Auth.confirmResetPassword

code used for reset:

Amplify.Publisher.create {
    print("going to call Amplify.Auth.resetPassword.")
    try await Amplify.Auth.resetPassword(for: email)
    print("Amplify.Auth.resetPassword called.")
}.sink {
    if case let .failure(authError) = $0 {
        print("Reset password failed with error \(authError)")
    }
} receiveValue: { resetResult in
    print("got reset result for Amplify.Auth.resetPassword: \(resetResult)")
}

code used for confirm:

Amplify.Publisher.create {
    print("going to call Amplify.Auth.confirmResetPassword.")
    try await Amplify.Auth.confirmResetPassword(for: email, with: newPassword,
                                                confirmationCode: verificationCode)
    print("Amplify.Auth.confirmResetPassword called.")
}.sink {
    if case let .failure(authError) = $0 {
        print("Reset password failed with error \(authError)")
    }
} receiveValue: {
    print("Password reset confirmed")
}

Expected behavior
once the request is completed - the sink should be triggered with either an error or a result, like it is explained in your documentation - https://docs.amplify.aws/lib/auth/password_management/q/platform/ios/
Amplify Framework Version
2.17.1 & 2.17.2
Amplify Categories
Auth
Dependency manager
Swift PM
Swift version
5.8, 5.9
CLI version
11.0.0 & 12.4.0
Xcode version
Xcode 15.0 Build version 15A240d, also happened before update to 15.0
Relevant log output
<details>
<summary>Log Messages</summary>

going to call Amplify.Auth.resetPassword.
2023-09-19T12:10:06+0300 info CognitoIdentityProviderClient : [Logging] Request: POST https:443
 Path: /
 Content-Length: 5884,
Content-Type: application/x-amz-json-1.1,
X-Amz-Target: AWSCognitoIdentityProviderService.ForgotPassword,
x-amz-user-agent: aws-sdk-swift/1.0,
User-Agent: aws-sdk-swift/1.0 api/cognito-identity-provider/1.0 os/iOS/17.0.0 lang/swift/5.9 lib/amplify-swift/2.17.2,
Host: cognito-idp.us-east-1.amazonaws.com
 Optional([])
2023-09-19T12:10:06+0300 info SerialExecutor : [Logging] Creating connection pool for Optional("https://cognito-idp.us-east-1.amazonaws.com/?")with max connections: 50
2023-09-19T12:10:06+0300 info CRTClientEngine : [Logging] Connection was acquired to: Optional("https://cognito-idp.us-east-1.amazonaws.com/?")
Amplify.Auth.resetPassword called.
-----------------------------
going to call Amplify.Auth.confirmResetPassword.
2023-09-19T12:11:23+0300 info CognitoIdentityProviderClient : [Logging] Request: POST https:443
 Path: /
 x-amz-user-agent: aws-sdk-swift/1.0,
User-Agent: aws-sdk-swift/1.0 api/cognito-identity-provider/1.0 os/iOS/17.0.0 lang/swift/5.9 lib/amplify-swift/2.17.2,
Host: cognito-idp.us-east-1.amazonaws.com,
X-Amz-Target: AWSCognitoIdentityProviderService.ConfirmForgotPassword,
Content-Type: application/x-amz-json-1.1,
Content-Length: 5934
 Optional([])
2023-09-19T12:11:23+0300 info SerialExecutor : [Logging] Creating connection pool for Optional("https://cognito-idp.us-east-1.amazonaws.com/?")with max connections: 50
2023-09-19T12:11:24+0300 info CRTClientEngine : [Logging] Connection was acquired to: Optional("https://cognito-idp.us-east-1.amazonaws.com/?")
Amplify.Auth.confirmResetPassword called.```
</details>

Is this a regression?
No
Regression additional context
No response
Platforms
iOS
OS Version
16.4, 17.0
Device
physical iPhone 12 as well as iPhone 14 & 15 simulators
Specific to simulators
no
Additional context

other auth operations in the app work fine and receive errors/result from the publisher using similar code - like sign in and sign out for the same user in same cognito pool

Hi @atierian,
tried using the code you provided and got the same behaviour - the reset password request was processed and i got an email with a verification code, but the sink wasn't triggered with any event - error or result.

that was the issue. it was missing and after i added .store() the functions work as expected.
Thanks @atierian


how to run the code ?

Hi! Thank you for your interest to this project.
After cloning this project, please first change line 45 in warehouse_test.py and line 17 in RLBrain.py to your own file path. Then run file warehouse_test.py to see the simulation.

I cloned your new code, and I found the same problem as follows：
File "c:\Users\efeff\Desktop\Warehouse_Robot_Path_Planning-master\RL_brain_Robo1.py", line 23, in init
f = open(RLBrain.FILEPATH + '\q_table1.txt', 'rb')
I cannot found this file in your repository, where is the q_table1.txt?

My bad. I just upload them. Thank you!

Thank you! one more question, if I want to retrain the Q-table, should I change the currentstate to state0 and then state1 so as to state2?

State 0 corresponds to training the map
State 1 corresponds to training the robots to return to the starting point
State 2 corresponds to training the robots to reach the target
These three states follow the three training steps in the flow chart in Readme.
Thanks!

I'm sorry, I am a little puzzled, then how to test the trained network?


Regarding the application of LoRA to the transformers

In the system diagram its shown as LoRA is applied for each transformer block. But inside the transformer block which layers are applied with LoRA according to your script provided? Is it Multi-Head Attention? or Feedforward? or both? Could you explain this.

create_qlora with heads accepts a LoraConfig.
In that one, you can specify target_modules to select for which parameters you want to train LoRA weights. Check here for some more info. A common choices for LLaMA based models is [q_proj, v_proj], e.g. the projection layers of query and value for the attention mechanism.
By default (when target_modules is None), transformer heads will train LoRA parameters for all linear layers. This is architecture-independently usually a reasonable choice that will include a few more layers than the query and value proejction, but not for example normalization parameters of a LayerNorm layer.
It can be useful to list all named paramters of your transformer head model to get a better understanding:
for name, param in model.named_parameters():
    print(name)

There you will see that added Lora Parameters which have Lora_A and Lora_B in their name.

Thanks a lot for the detailed explanation! :)


Rename ruby jenkinsfile and update runner tag

Signed-off-by: Peter Zhu<EMAIL_ADDRESS>Description
Rename ruby jenkinsfile and update runner tag
Issues Resolved
https://github.com/opensearch-project/opensearch-build/issues/2360
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #2371 (47cabbc) into main (5c7903d) will increase coverage by 5.38%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #2371      +/-   ##
=============================================
+ Coverage     94.61%   100.00%   +5.38%
=============================================
  Files           209         6     -203
  Lines          4309       105    -4204
  Branches         29        19      -10
=============================================
- Hits           4077       105    -3972
+ Misses          226         0     -226
+ Partials          6         0       -6

Impacted Files
Coverage Δ

..._workflow/integ_test/service_termination_result.py

src/manifests_workflow/component_opensearch_min.py

src/test_workflow/bwc_test/bwc_test_suite.py

src/run_manifests.py

src/run_build.py

tests/jenkins/jobs/Messages_Jenkinsfile

src/run_integ_test.py

src/ci_workflow/ci_check_list.py

...est_workflow/test_result/test_component_results.py

...rc/test_workflow/test_result/test_suite_results.py

... and 193 more

Help us with your feedback. Take ten seconds to tell us how you rate us.


Raw Json property on PocketItem

Just updated to NuGet package version <IP_ADDRESS> and it looks like the Json property off of the PocketItem object was removed.  Master branch shows it as still there, but the simple branch shows it as missing.
Is this version from the simple branch?  If the change was reverted, can I get an update to the NuGet package with the return of the property?

Hi!
Best thing for now is to get the previous nuget package.
Don't know why I did this, was not on purpose. I will fix this in the next days!

Thank you, @ceee!

Any update on re-adding this?

Any update on re-adding this?

The current version includes an AfterRequest action where I pass the raw JSON response, so you can handle the response yourself in case you want to. (https://github.com/ceee/PocketSharp/commit/8248f90b5b97176a8b3fc5117f56b2fae730dc71)
The JSON Property has been completely removed from PocketItem.
This change is not on NuGet yet, but will be introduced as a breaking change in the next version.


UI: undo/redo functionalty

Right now no changes propagated in the data model for this shortcuts.

Right now undo/redo brings inconsistent data model state while collaborative work thus It should be disabled.


ROUTER - failing a test in the level0 "fifo_arb_back_pressure"

Failing this tests in the level0
Please review the tests, logs, waves and see if this is an RTL issue or verification issue
@yeonatanPerelman @ShmuelSfez
python build.py -dut 'router' -debug -tests fifo_arb_back_pressure -hw -sim -params ' -gV_REQUESTS=10 -gV_FIFO_DEPTH=4 -gV_NUM_FIFO=4 -gV_MAX_DELAY=5 -gV_BACK_PRESSURE=1'

Please note:
We had an RTL bug in the FIFO that we fixed last commit & PR:
We had the full indication set before the FIFO was actually full - we fixed it and its already In main

i just pushed a fix.


Error 500, niet installeren werkt gewoon totaal niet

Expected behavior
Betaalmethode aan klanten aanbieden
Actual behavior
Betaalmethode wordt aangeboden maar geeft direct daarop een error 500 (betreft alle versie 4.0
Steps to reproduce the behavior
Versie V4.0.9 installeren of V4.0.8 en alles wijst vanzelf
Environment

PrestaShop version: <IP_ADDRESS>
Mollie module version: 4.0.9
PHP version: 7.2.34

Hey @Goedkoopsteverf sorry to hear about this, but could you be more detailed?
Could you provide us the screen-capture reproduction step by step? Or at least more detailed steps with some graphics?
The cause of this could be multiple, maybe cache, maybe the Prestashop theme or maybe something else.
Best Regards.

This is simple as our client has placed the items in the cart and choose wich paymentmethode will be used by clicking te conformation the nex screen apears with a error 500, this is not logged in the Prestrashop Logfiles.

@Goedkoopsteverf ok, thanks for your information, did the client delete the old version from the Prestashop and all the folder, and do fresh 4.0.9 install?
Best Regards.

@Goedkoopsteverf thanks for your feedback, we are checking this out with the team.
We will update you with the notification about the fix.
Best Regards.

Same problem here. Also a 500 error when the payment is started.
When debugging Prestashop I get:
Can't load Order status (id_order_state: 0)
at line 244 in file classes/PaymentModule.php
Apparently the configuration value cannot be loaded properly, so it is passed as 0 and then things go wrong.
In other words, the id_order_state is passed incorrectly. I was able to fix it temporarily to change the code below:
\modules\mollie\controllers\front\payment.php
$this->module->validateOrder(
            (int) $cartId,
	    (int) Configuration::get(Mollie\Config\Config::STATUS_MOLLIE_AWAITING),
            $totalPrice->toPrecision(2),
            isset(Mollie\Config\Config::$methods[$apiPayment->method]) ? Mollie\Config\Config::$methods[$method] : $this->module->name,
            null,
            $extraVars,
            null,
            false,
            $secureKey
        );

into:
        $this->module->validateOrder(
            (int) $cartId,
	    1,    // Temporary solution for the order status
            $totalPrice->toPrecision(2),
            isset(Mollie\Config\Config::$methods[$apiPayment->method]) ? Mollie\Config\Config::$methods[$method] : $this->module->name,
            null,
            $extraVars,
            null,
            false,
            $secureKey
        );

In /modules/mollie/src/config/Config/Config.php I see:
const STATUS_MOLLIE_AWAITING = 'mollie_awaiting';
We are using Prestashop <IP_ADDRESS>
Upgrade Mollie module from 3.4.5 to 4.0.9
Hopefully you can find the cause and make a nice solution for it. Best Regards.

I have similar issue .
Prestashop <IP_ADDRESS>
Logs show following error.
PaymentModule::validateOrder - Order Status cannot be loaded
Same issue in 4.0.8

Hey guys,
After you upgrade the module from 3.4.5 to 4.0.9 or 4.0.8, you should always reset the module additionally.
We reproduced that case on localhosts and test environments.
We have backloged this, we will check what best we can do in order to solve this not user friendly solution.
Best Regards.

Advice don't install won"t work even what the develloper said, seach for a other PSP with betere payment plugins for Prestashop


Run by line does not work.

Environment data
Build of Visual Studio Code - Insiders: 1.68.0-insider(user setup)
Build of python-Pre-Release: v2022.7.11511005
Build of jupyter-Pre-Release: v2022.5.1001532360
Build of pylance: v2022.5.3
Build of jupyter Notebook Renderers: v1.0.8
Build of jupyter Keymap: v1.0.0

Steps to reproduce:

Opt into the custom editor experiment (not yet supported in native notebooks)
Open a new blank Jupyter notebook.
Add the following code to a cell:

def bar():
    for i in range(5):
         print(i)

def foo():
    bar()

foo()
print('Done')

Press F10 to step through the code in the cell line-by-line.

Expected behaviour
Verify that it first steps over function definitions for foo() and bar(), then steps into the function definitions after foo() is called, then steps back out when the loop in bar() finishes and prints 'Done'.
Actual behaviour
No response when press F10
Jupyter Log

This seems kind of important. Run by line isn't working.


HDH-297-add-math-packages-to-ds

Thanks for contributing to the Harness Developer Hub! Our code owners will review your submission.
Description

Please describe your changes: __________________________________
Jira/GitHub Issue numbers (if any): ______________________________
Preview links/images (Internal contributors only): _____

PR lifecycle
We aim to merge PRs within one week or less, but delays happen sometimes.
If your PR is open longer than two weeks without any human activity, please tag a code owner in a comment.
PRs must meet these requirements to be merged:

[ ] Successful preview build.
[ ] Code owner review.
[ ] No merge conflicts.
[ ] Release notes/new features docs: Feature/version released to at least one prod environment.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for working on @rohanmaharjan100. Looks like a build error. Also we can add the Math Example to our Sandbox MD file?
https://github.com/harness/developer-hub/blob/main/docs/hdh/hdh-docusaurus-sandbox.md
Docusaurus could not load module at path "/harness/docusaurus.config.js"

Cause: ParseError: Unexpected token, expected ","

 /harness/docusaurus.config.js:277:15

    at loadFreshModule (/harness/node_modules/@docusaurus/utils/lib/moduleUtils.js:36:15)

    at loadSiteConfig (/harness/node_modules/@docusaurus/core/lib/server/config.js:35:62)

    ... 3 lines matching cause stack trace ...

    at async Command.build (/harness/node_modules/@docusaurus/core/lib/commands/build.js:59:21) {

  [cause]: Error: ParseError: Unexpected token, expected ","

   /harness/docusaurus.config.js:277:15

      at loadFreshModule (/harness/node_modules/@docusaurus/utils/lib/moduleUtils.js:33:16)

@rohanmaharjan100 was there any yarn or package changes missing in the commit? Maybe why the build is failing.
https://docusaurus.io/docs/markdown-features/math-equations


Is There a Way to Request an Access Token?

Hello:
I've been reading a ton of the questions/answers and I still can't determine 100% whether I can login my user (by using their email/password) and request an access token that I can use for subsequent requests and sessions.
I don't want to and its not good practice by any stretch to store the users' usernames and passwords in my system.
I've used the Office 365 REST API and have been able to request the access token and use it.  I'm just trying to duplicate this with the Exchange API.
If it helps here is the scope of what I'm trying to do:  I have a web-app that needs to be read/write things to the users' calendar and so what I would like to allow the user to authenticate using their username/password with my app receiving the access token that can be used for future queries.
Thanks for your help.

This library doesn't currently support this method of authentication, as it's only available with Exchange Online. I'm would be open to any submitted patches, but I do not have this on my radar at this time.

Okay understood.  I guess I'm just trying to figure out how other people do this then?  I surely hope they're not storing people's email addresses and passwords or forcing them to login each and every time they open an app.

Hi @chodges
May I suggest you try my own ews library, garethp/php-ews,  which does in fact come with support for fetching an access token and using that as authentication

Hi @Garethp
Thanks yes I know of your library and I took a look at it.  I think the problem that I'm facing is I'm attempting to connect to a hosted Exchange server that just does not have the token access method available.

Hm... Is it the latest exchange version? What version are you running? And is your authentication maybe backed by Azure?

HI @Garethp
I'm not sure which version of exchange it is.  I'm building a custom CRM that needs to connect to people's Outlook to add/edit calendar events.  So in reality I'm not going to always know which version of Exchange it is that they are running or if its backed by Azure.
I do know that this a third-party Exchange server running on their IT company's infrastructure so its not a direct Microsoft hosted account.

Then unfortunately it's not possible. OAuth is the only token-based authentication that Exchange Provides, and that's only available when the authentication is backed by Azure Directory Services. Exchange does also provide NTLM Authentication, so that people who are already logged in to their domain account don't have to log in again, but I've been struggling to find a way to make this work in PHP.
Your best option may either be to have the user give their password with each login, or have a master account that has impersonation rights, and store the password for that account in a secure manner

@Garethp I wouldn't have a problem using another technology (e.g. Javascript) to enable the user to log in and then grab the token but I haven't come across a way to do even that.

Created a feature request at jamesiarmes/php-ntlm#2 for this.

@Garethp , I have did some small changes in your ExchangeWebServicesAuth.php's fromUsernameAndPassword function as below. It worked as expected from my locahost but i am yet to check in production server.
replace the existing return statement with below one.
    return array(
        'curl' => array(
            CURLOPT_HTTPAUTH => CURLAUTH_NTLM,
            CURLOPT_UNRESTRICTED_AUTH => true,
            CURLOPT_USERPWD => ":"
        )
    );

I have also changed the Mail API::withUsernameAndPassword parameter as below.
$server = 'xxx.bbb.com'
$username = '';
$password = '';
$api = MailAPI::withUsernameAndPassword($server, $username, $password);
I was able to see the sent mail in my actual display name.  I think the empty  ":"
parameter for CURLOPT_USERPWD has taken the current user.
Reference Link:
https://stackoverflow.com/questions/23522503/can-i-bypass-the-username-pw-using-curl-with-iis-windows-authentication-in-php

@Garethp have you considered merging your improvements back into @jamesiarmes's repo? There is another fork that also has OAuth support: https://github.com/barracudanetworks/php-ews. I am trying to figure out if it is worth trying to rebase that repo and create a PR.

@ablyler I had a PR open at the beginning, but at that point James was busy with other things. In the end our projects grew too far apart to really merge in. We discussed it, but our libraries are trying to solve separate problems and achieve separate goals. At this point my library won't be easily merged in.
I can't really comment on whether or not it would be worth it for you to change what repo you're using


Conventions and guidelines

As the project is growing I think it would be good to introduce some conventions and guidelines to keep everything a bit more uniform.
For example:

Coding guidelines
Merge strategy (e.g. --ff-only)
Conventions of git messages

It's possible to add verification to travis-ci that would automate the process.
Any thoughts?

I think that's a great idea.
Also I think we should make a rule that even us maintainers should only use PRs to change the codebase. Right now this is already happening for bigger features and stuff, but I think we should do this for all changes. Like this we would have a dedicated space for each change, it's easier for everyone to keep track or what's happening and why and this also allows us to group a set of changes together. This can also be used to make sure that no commit fails a travis build.
For coding guidelines it would probably be good to add a pre commit hook that automatically formats the code to conform to certain style guidelines.
I don't think having many coding rules is wise because it would add a lot of overhead for first-time contributors and it probably wouldn't help the readability of the code that much.
Can --ff-only be forced in a github PR or would we have to merge PRs on the command line?

Yeah I totally agree about forcing pull-requests for everyone. I think that can be fixed by changing the repository settings. I will have a look.
I'm generally against applying changes automatically when performing an action (e.g. reformatting the code upon commit) but a travis check using for example clang-format could hint the contributor that the committed code needs to be changed.
I agree that it's not a good idea to add alot of overhead for first-timers but some restrictions is, in my opinion, needed to maintain the quality. A clang-format check does of course not ensure the quality of the contributed code but it's a start to keep the code uniform.

I removed the ability to merge pull-requests using --no-ff.

There's no way to block changes, done by contributors, to the default branch from what I can see in the settings. But we are still a low number of contributors so for now I guess we could agree that we create a PR for all changes. And of course we cannot merge the PR ourselves.

Yeah maybe we could add a separate travis check just for clang-format
Just agreeing on PRs only will probably suffice. For small PRs, I think that we could allow people to merge their own PRs, if it really messes things up, it's easy to undo the whole PR at once.
While I have you here, can you add us to the xpp and i3ipcpp repos?

Regarding small PR's: I still think we should have a requirement of at least 1 approval (unfortunately I don't think we can enforce it by a GitHub setting). After that you could of course merge it yourself.
Yeah, of course, I'll add you as contributors there as well.

I have added all polybar contributors to the lib forks (xpp, i3icpp).

Should we have a todo list of some kind? We could possibly create a kanban board under Projects and manage stuff there.

We could also open a PR which contains those guidelines written down and we could then track everything over there. What do you think would be better?

I've been considering using Projects as a centralized location to manage feature requests/enhancements, so we could possibly include this over there, and convert it into a PR when we're ready.
Should this be under CONTRIBUTING.md?

Having feature requests in a centralized location will certainly help us keep track of them better and not let them slip through the cracks. How would discussions work in a setup like that? Can we do that inside the Project or would still need to use an issue or PR for this?

Should this be under CONTRIBUTING.md?

What do you mean by "this"?

From what I've seen, if a card doesn't have an issue yet, there is an option to convert it into an issue, so discussions should occur there. Existing PRs and issues can be added as-is.
By this, I mean "conventions and guidelines", as in listing all of them within CONTRIBUTING.md.

I second this.


Move to GitHub action for releases

Introduces GitHub Actions for deploying the changes to wordpress.org instead of
our existing manual process.
In it's current form, the TL;DR of the manual process looks like this (full details).

Update backend/frontend repositories with whatever changes are required.
Update README with version bumps and CHANGELOG entries.
Draft a new GitHub release (without artifacts).
Clean up the repository and build the intended structure for SVN.
Commit, tag and push the SVN changes to wordpress.org
Update the GitHub.

As the linked documentation shows, this is a pretty complex workflow with
plenty of places that people could fall into the abyss without a great recovery
plan.
To assist with automation and consistency, I've ported majority of the steps
into a GitHub Action that is triggered on release. The updated workflow is now:

Update the backend/frontend repositories.
Update README with version bumps and CHANGELOG entries.
Create a new GitHub release via the UI.
Profit ✨

The SVN interactions, repository packaging and directory structure munging is
now handled behind the scenes. The release action will handle generating the
correct structure, committing it and uploading the generated artifact back to
the GitHub Action.

Thank's for working on this, I'm looking forward to test it out, once merged.

Thank's for working on this, I'm looking forward to test it out, once merged.


feat(networkfirewall): add new check networkfirewall_policy_default_action_drop_forward

Context
In a network firewall policy, stateless actions are applied to packets without maintaining session state. This is crucial for efficiently handling high volumes of traffic, where the default action determines whether packets are allowed, dropped, or forwarded. Having the correct default action helps prevent unauthorized or unintended traffic from passing through the network firewall, which is especially critical in maintaining a secure network configuration.
Description
This check ensures that the default stateless action for full packets in AWS Network Firewall policies is set to either drop or forward. It evaluates firewall configurations to prevent the default setting from being "pass," which could allow unintended traffic through the firewall. A proper configuration is essential to control how the firewall handles network traffic in a Virtual Private Cloud (VPC). If the default action is set to pass, the check fails, as this could expose the network to security risks by allowing unfiltered traffic.
Checklist

Are there new checks included in this PR? Yes.

If so, do we need to update permissions for the provider? Please review this carefully.

[x] Review if the code is being covered by tests.
[x] Review if code is being documented following this specification https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings
[x] Review if backport is needed.

License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The name of this check is also open to suggestions as I'm not able to figure out a better one. Other ideas are:

networkfirewall_policy_default_stateless_action_secure_full_packets
networkfirewall_policy_default_stateless_action_secure_full_packets
networkfirewall_policy_drop_or_forward_full_packets


Update derive_var_disposition_status.R to have more meaningful argument

Currently the argument uses x - but I think it should have a more meaningful name

@bms63 what name would you suggested other than 'x' , maybe 'disposition_var' since a disposition variable is expected for that format function. That said, is this still necessary? It looks like the function derive_var_disposition_status  which uses this format utility is depreciated as of last month. @amsmith214


Blazing fast ChunkDataStreams

Fixes #256
The previous API did not specify whether ChunkIterator would be blocking on the main thread and provided little ways to read specific chunk data.
This adds the ability to get a specific Chunks data via asynchronous ListenableFuture and aids in refining the asynchronous nature of ChunkDataStream.

Obligatory #420 'Blazing'
This PR is quite a nice, I don't see any issues with it.

I like this quite a bit. This solves completely what the Iterator was half-solving.

The comments at least address the major possible issue with doing this outside of the main thread (data is essentially a 'snapshot')
I'm not quite sure what the point of using Future<> is if you're telling people not to wait for it though.

I'm not quite sure what the point of using Future<> is if you're telling people not to wait for it though.

To pass the future to an async thread that can wait for the future to complete. From the JDK Javadocs:

The result can only be retrieved using method get when the computation has completed, blocking if necessary until it is ready.

We definitely do not want to block the main thread when we get from the future. The idea is simply to just pass off the future to another thread and wait on that thread.

Do you have some test-plugin code that demonstrates the capability?
Date: Wed, 28 Jan 2015 11:07:23 -0800
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Subject: Re: [SpongeAPI] Blazing fast ChunkDataStreams (#420)
I'm not quite sure what the point of using Future<> is if you're telling people not to wait for it though.
To pass the future to an async thread that can wait for the future to complete. From the JDK Javadocs:
The result can only be retrieved using method get when the computation has completed, blocking if necessary until it is ready.
We definitely do not want to block the main thread when we get from the future. The idea is simply to just pass off the future to another thread and wait on that thread.
—
Reply to this email directly or view it on GitHub.
	 	   		  =

Perhaps use guava's ListenableFuture instead? Seems a bit more friendly for the "preferred" async callback scenario. Otherwise you're going to have code that will end up waiting for the future on the main thread at some point (otherwise why bother calling the method?)

Otherwise you're going to have code that will end up waiting for the future on the main thread at some point (otherwise why bother calling the method?)

Not if the developer is using the future correctly. It's simple to pass in a Future to a new thread and wait for it in said thread, avoiding blocking the main thread.

Perhaps use guava's ListenableFuture instead? Seems a bit more friendly for the "preferred" async callback scenario.

Sure, we could use that instead, registering callbacks is a nice idea.
Thoughts @SpongePowered/developers ?

Async event handlers? Crazy but thought I'd mention it.

@Zidane I believe this is ready for merging.

:+1:


rename asm feature to arch; asm! -> llvm.arm.*

replace the asm! implementation with LLVM intrinsics; this lets the optimizer
unroll the LL/SC loop

bors r+


Requests

-BlockBot.
-Door spammer
-98 Nade

The blockbot and door spammer are in the pull request #739

The blockbot and door spammer are in the pull request #739

sorry I'm pretty dumb all of that is very confusing 2 me

door spammer already added check pull requests

add them urself ._.

Runboost helper please :(


Add-feature-custorm-response-body

Hi! Nice to meet you.
I want to contribute to this repository.
what

Added the ability to specify a custom response body.

why

Because I wanted to implement the maintenance window feature.

/test all

@vampire-yuta tests are failing. I think the dynamic will fix this.

Please check it!
If you need test,please rerun.

Hi. Test was passed.
How about this?? 👀

/test all

/rebuild-readme

/test readme

/test all

looks good!

Is there anything I can do to help get this feature added to an upcoming release?

/test all

/test all

@vampire-yuta sorry, we're running into some issues with our CI. Can you run the following locally and push the changes?
make init
make github/init
make readme


Changes to make it work with client properly

Hi guys, I create a seeds.rb with all we need to fill fast our DB and render components easilly.
Passes all validation models, and creates only one reservation per user and per course.

4 reservations for user
5 users
20 courses

@NelsonAraujo93 @kensteph @biruk-bereka Im goin to close this PR to avoid confusions


⚠ Site ANTS has degraded performance

In 038318b, Site ANTS (https://ants.gouv.fr) experienced degraded performance:

HTTP code: 200
Response time: 6685 ms

Resolved: Site ANTS performance has improved in e12e821.


Password rotation possibility

Hi,
Is there any possibility to change the password but not touch already encrypted values? The requirement is to rotate passwords once in 2 months, and I cannot find a way how to do it without already encrypted data re-encryption. It will be really helpful to have a master password and a secondary password, so we can put them and Jasypt can automatically handle it.

I’m confused, change on any secret… encryption key or encrypted secret will require you to change the values in the property files…

@ulisesbocchio Let me rephrase my problem.
We are using Jasypt to keep the data encrypted in the DB (Hibernate). So we set a password let's say 123, and we have 10 records in the DB with encrypted value through Jasypt. Now, I want to change the password from 123 to 234, but the existing records in the DB were encrypted with the password 123, correct? So my question was, is there any possibility for password rotation without touching already encrypted values in the DB, or do we need to write a job or so which will do the rest? e.g. get the encrypted data form the DB using an old password, encrypt with a new password, and put back. Hope I was clear now, thanks!

Ah, the answer is yes, but not for free :)
You'd have to implement depending on the level of sophistication needed either a:

Custom Encryptor
Custom Property Detector
Custom Property Resolver

Or all of the above....
the fastest way is probably defining custom composite StringEncryptor, this string encryptor would be bootstrapped from multiple keyed string encryptors. For instance:
jasypt:
    encryptor:
      encryptors:
          - key: old
             default: true
             privateKeyFormat: PEM
             privateKeyLocation: classpath:private_key_old.pem
          - key: new
             privateKeyFormat: PEM
             privateKeyLocation: classpath:private_key_new.pem

You'd figure out the code, but the trick here would be to also "enhance" the encrypt and decrypt methods:
    private static class KeyedMessage {
        public static parse(String value) {
             // split incoming msg expecting a char separator like @ or whatever, for instance: `old@uaosdifuosidfu`
             return new KeyedMessage(key, message);
        }

        public String toString() {
               return this.key + "@" + this.message;
        }
    }

    private StringEncryptor getDelegate(String key) {
        // find own encryptors by key, or if key === null, use default one.
    }

    @Override
    public String encrypt(String message) {
        KeyedMessage msg = KeyedMessage.parse(message);
        String encrypted = this.getDelegate(msg.getKey()).encrypt(msg.getMessage());
        return new KeyedMessage(msg.getKey(), encrypted).toString();
    }

    @Override
    public String decrypt(String encryptedMessage) {
        KeyedMessage msg = KeyedMessage.parse(message);
        return this.getDelegate(msg).decrypt(msg.getMessage());
    }

then essentially you can add/remove encryptors to your list via config, and change the encrypted values in the DB at your own pace, so long as you don't remove the encryptor config before you remove the last encrypted password with that encryptor.

Thanks @ulisesbocchio , I will check the solution and repost it here.


Convert QA Book from TeX to Markdown

@KatieWoe, I'm going to keep track of the QA Book conversion here.
To Do List:

[x] Skeleton / Outline
[x] Meat / Content
[ ] Review with Katie
[ ] Make Changes
[ ] Upload QA Book to GitHub
[ ] Add Links in Templates

@KatieWoe, ready for review when you are!


Use structured logging for the framework logs

The framework logs some events (mostly exceptions), we should use structured logs for them.

https://cloud.google.com/functions/docs/monitoring/logging#writing_structured_logs


CVE-2022-28948 update

Github security advisory
update stretchr/testify to 1.7.2 to mitigate go-yaml.v3 issue.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Codecov Report

Merging #446 (5f9cfb6) into main (0dae564) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #446   +/-   ##
=======================================
  Coverage   72.90%   72.90%
=======================================
  Files          19       19
  Lines        1085     1085
=======================================
  Hits          791      791
  Misses        225      225
  Partials       69       69

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0dae564...5f9cfb6. Read the comment docs.

🤷♂ Did this trigger a warning in a consuming package you have? The dependency only comes in for running this package's unit tests, so it'd interesting & unexpected if the vulnerable dependency was being transitively picked up in the consumer.

🤷♂ Did this trigger a warning in a consuming package you have? The dependency only comes in for running this package's unit tests, so it'd interesting & unexpected if the vulnerable dependency was being transitively picked up in the consumer.

bryan@air yolo % cat main.go
package main

import (
	"github.com/aws/aws-lambda-go/lambda"
)

func main() {
	lambda.Start(func() {})
}

bryan@air yolo % cat go.mod
module yolo

go 1.18

require github.com/aws/aws-lambda-go v1.32.0

bryan@air yolo % cat go.sum
github.com/aws/aws-lambda-go v1.32.0 h1:i8MflawW1hoyYp85GMH7LhvAs4cqzL7LOS6fSv8l2KM=
github.com/aws/aws-lambda-go v1.32.0/go.mod h1:IF5Q7wj4VyZyUFnZ54IQqeWtctHQ9tz+KhcbDenr220=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ=

ahh, must be because of the gitHub.com/aws/aws-lambda-go/test package being in the repo
https://github.com/aws/aws-lambda-go/blob/main/events/test/jsoncompare.go#L8


Fix override issue in mod_ztportfolio

Can not chose multile views when override

Adding a litte code in mod_ztportfolio.xml


[BUG] Multimatch on fields with distincts synonym analyzers does not work properly

Describe the bug
I stumbled on an edge case when mixing different fields with search-time synonyms analyzers in a multimatch query. In the case described below we can see in the profiling information that the multimatch query is rewrite into an incorrect DisjunctionMaxQuery that targets the same field multiple times.
This bug is reproductible in 2.6.0 and 2.7.0
To Reproduce
Here is a minimal scenario  to reproduce the bug
curl -X PUT localhost:9200/synonyms -d '{
  "aliases": {},
  "mappings": {
    "dynamic": false,
    "properties": {
      "text": {
        "search_analyzer": "synonym_1",
        "type": "text"
      },
      "title": {
        "search_analyzer": "synonym_2",
        "type": "text"
      }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "synonym_1": {
          "filter": [
            "synonyms"
          ],
          "tokenizer": "standard"
        },
        "synonym_2": {
          "filter": [
            "synonyms"
          ],
          "tokenizer": "standard"
        }
      },
      "filter": {
        "synonyms": {
          "lenient": "true",
          "synonyms": [
            "term1, term2"
          ],
          "type": "synonym_graph"
        }
      }
    }
  }
}'

curl -X POST localhost:9200/synonyms/_doc/test_doc -d '{
  "title": "term1"
}'

curl -X POST localhost:9200/synonyms/_search -d '{
  "profile": "true",
  "from": 0,
  "query": {
    "multi_match": {
      "boost": 1,
      "fields": [
        "text^1.0",
        "title^1.0"
      ],
      "query": "term1",
      "type": "best_fields"
    }
  }
}'

This request does not match our document and in the profiling we can see this :
"profile": {
    "shards": [
      {
        "id": "[GaBuyLhQSyalJSCmyrTaIA][synonyms][0]",
        "inbound_network_time_in_millis": 0,
        "outbound_network_time_in_millis": 0,
        "searches": [
          {
            "query": [
              {
                "type": "DisjunctionMaxQuery",
                "description": **"(Synonym(text:term1 text:term2) | Synonym(text:term1 text:term2))",**

The text field is targeted twice and the title field is not used in the query.
Note that in version 2.5.0 this works correctly and the query generated is :
 "type" : "DisjunctionMaxQuery",
 "description" : "(Synonym(title:term1 title:term2) | Synonym(text:term1 text:term2))",

Also note that if we have more terms in the query it works fine :
curl -X POST localhost:9200/synonyms/_search -d '{
  "profile": "true",
  "from": 0,
  "query": {
    "multi_match": {
      "boost": 1,
      "fields": [
        "text^1.0",
        "title^1.0"
      ],
      "query": "term1 term3 ",
      "type": "best_fields"
    }
  }
}'

=>

"type": "DisjunctionMaxQuery",
"description": "((Synonym(title:term1 title:term2) title:term3) | (Synonym(text:term1 text:term2) text:term3))",

Expected behavior
The multimatch query should targets both fields
Plugins
No plugins
Host/Environment (please complete the following information):

OS: Docker
Version : 2.6.0

I've added a draft PR in #10148. It matches the exact request that is being made here, although I am not a 100% sure yet if this issue is with multi_match or with the synonym analyzer.
I'm having issues getting the test case to pass, I keep encountering this:
 2> java.lang.IllegalArgumentException: Unknown filter type [synonym] for [synonyms]
        at __randomizedtesting.SeedInfo.seed([91D695E83F6A921B:84D55249764E76E3]:0)
        at org.opensearch.index.analysis.AnalysisRegistry.getAnalysisProvider(AnalysisRegistry.java:565)
        at org.opensearch.index.analysis.AnalysisRegistry.buildMapping(AnalysisRegistry.java:516)
        at org.opensearch.index.analysis.AnalysisRegistry.buildTokenFilterFactories(AnalysisRegistry.java:336)
        at org.opensearch.index.analysis.AnalysisRegistry.build(AnalysisRegistry.java:239)
        at org.opensearch.index.IndexModule.newIndexService(IndexModule.java:628)
        at org.opensearch.indices.IndicesService.createIndexService(IndicesService.java:839)
        at org.opensearch.indices.IndicesService.withTempIndexService(IndicesService.java:782)
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:479)
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:584)
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:441)
        at org.opensearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:448)
        at org.opensearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:354)
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874)
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204)
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.lang.Thread.run(Thread.java:1623)

Looking at other tests that make use of search analyzers in settings, looks like I'm not doing anything too different but I can't seem to get out of this issue -- https://github.com/opensearch-project/OpenSearch/blob/0c4e95095e4d44540ea8a90aa5ca55fea1248366/server/src/internalClusterTest/java/org/opensearch/search/suggest/CompletionSuggestSearchIT.java#L1090
I've also verified that this issue does not occur in 2.5 and shows up in 2.6. Looking at the diff, I don't see any changes to the MultiMatchQuery or the SynonymAnalyzer, although I'm not entirely sure I'm looking at the right places -- https://github.com/opensearch-project/OpenSearch/blame/750901a2d1304961070563a66313fa34c03ec779/server/src/main/java/org/opensearch/index/search/MultiMatchQuery.java#L221
There was a lucene version upgrade that could've caused this issue. Any thoughts @msfroh @rishabhmaurya

For information, this issue is no longer reproductible in version 2.10 of opensearch ( I did not tried the 2.9 version ).

Thanks @msfroh for pointing the right place!!! It's fixed by Lucene at 9.6 here, I can upgrade the lucene version at 2.6  and 2.7  to fix the issue for the multi-match query not working property, shall I ?

I can upgrade the lucene version at 2.6 and 2.7 to fix the issue for the multi-match query not working property, shall I ?

We don't upgrade the Lucene version on past releases.
In this case, I believe the correct course is just to say "This is a known bug in 2.6 and 2.7, but we know it is fixed in 2.8 and later".

Let's get https://github.com/opensearch-project/OpenSearch/pull/10148 to pass and close this with it? @harshavamsi


mixing style latents

with the previous methods I could replace layers
8 to 18 from one latent to another latent to change the style of latente1.
from 0 to 8 is the pose of the face and from 8 to 18 is lighting and texture of the face
latent 1 = 1 18 512
latent 2 = 1 18 512
mix_laten = latent1 [0: 8] + latent2 [8:18]
But this generator will require weights in addition to latent, I don't know how to "split" weights_deltas for mix styles, how can I mix styles?

Yea, although we didn't try it out in the paper, I do believe that mixing will still work here.
If you take a look at the project page, you can see that there are various animations showing interpolations between images. I think that this interpolation is similar to style mixing.
What we did there was interpolation between both the latent code and the weights.
For example,
cur_vec = vectors_b * alpha + vectors_a * (1 - alpha)
cur_weight_deltas = interpolate_weight_deltas(weights_deltas_a, weights_deltas_b, alpha)
res = net.decoder([cur_vec],
                  weights_deltas=cur_weight_deltas,
                  randomize_noise=False,
                  input_is_latent=True)[0]

where alpha is your blending factor.
So with style mixing, I think what you could do is try doing something like:
mix_latent = latent1[0: 8] + latent2[8:18]
mix_weight_deltas = deltas1[0:8] + deltas2[0:8]

I am not sure this would work, but this is what I would try :)