Time filter no longer working with latest version [Kibana 7.9.2 - ET 1.10.0]

Time filter in Kibana is no longer working with latest version of Enhanced Table.
Always all results will be shown regardless of the settings in the time filter.
Enhanced Table:
13059 results shown, not correct for "Today" time filter

Kibana Data Table:
327 results shown

Discover:
327 results shown

Hi @hoberhol,
I just released enhanced-table 1.10.1, with this issue fixed:
https://github.com/fbaligand/kibana-enhanced-table/releases/tag/v1.10.1

Hi @fbaligand,
I tried using 1.10.1 today, but whenever I try to create a visualisation with an enhanced table I get the following error:
ChunkLoadError: Loading chunk 9 failed. (error: http://localhost:5601/33984/bundles/plugin/enhancedTable/9.plugin.js)     at Function.requireEnsure [as e] (http://localhost:5601/33984/bundles/plugin/enhancedTable/enhancedTable.plugin.js:106:26)     at http://localhost:5601/33984/bundles/plugin/enhancedTable/enhancedTable.plugin.js:191502:129     at http://localhost:5601/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:422:105001     at xs (http://localhost:5601/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:422:105150)     at ml (http://localhost:5601/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:422:90018)     at hl (http://localhost:5601/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:422:89943)     at ol (http://localhost:5601/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:422:87291)     at http://localhost:5601/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:422:45733     at t.unstable_runWithPriority (http://localhost:5601/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:430:3462)     at Hi (http://localhost:5601/33984/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:422:45442)
I am using Kibana v7.9.2.
Any pointers as to what might be causing this?

Hi @fbaligand,
Thanks a lot for the fast bugfix. I tested the new version and everything is now working just perfect!
@WizBangCrash: I cannot reproduce your issue. On my side I can create a new visualisation with an enhanced table without any problem.
I'm using 1.10.1 with 7.9.2 as you.
Have a nice evening.
Regards,
Hans

@hoberhol
Happy to see that you confirm the bugfix!
@WizBangCrash
I invite you to follow steps indicated in Troubleshooting section:
https://github.com/fbaligand/kibana-enhanced-table#troubleshooting

@fbaligand
FYI. I rebuilt my docker image with the --no-cache option and I no longer have the error.
Thanks for the latest version :-)

Great!
Happy to see that it finally works, and thanks for the feedback.
It's nice to know for other users that can have the same issue.


Add _Navigation.sass

Hello :)
I want to include navbar in milligram, so I copy navbar of http://milligram.github.io/ .
I like this navbar style:)
I recommend navbar because bootstrap is navbar.(now is navigation)
I want to implement navbar-right, navbar-left, navbar-fixedand etc... in the future.

Hi @abouthiroppy
The idea is great but my intention is to keep the Milligram as light as possible, supporting the feature minimum necessary to initialize a project. BTW, what you are proposing would be a very specific component. I'm closing your request and appreciate their efforts to improve this project.


Typescript 2.9 breaks: Error TS2536: Type 'P' cannot be used to index type 'Si'

Code to reproduce the issue:
Use @cycle/run package with typescript ^2.9
Actual behavior:
[12:44:56] [bro] TypeScript error
TypeScript error: node_modules/@cycle/run/lib/cjs/types.d.ts(32,36): Error TS2536: Type 'P' cannot be used to index type 'Si'.
[12:44:56] [bro] TypeScript error
TypeScript error: node_modules/@cycle/run/lib/cjs/types.d.ts(32,43): Error TS2536: Type 'P' cannot be used to index type 'So'.

Versions of packages used:
"@cycle/dom": "^20.4.0",
"@cycle/isolate": "^3.3.0",
"@cycle/run": "^4.1.0",

TypeScript once in a while issues breaking changes under the minor number (e.g. 2.9.1) which is against SemVer. We will aim to update cycle/run with TS 2.9, but note that you are using an incompatible TS. Prefer using ~2.8.0 not ^2.8.0 as the version in your dependencies.

I never actually saw the Drivers type until now that it bit me. My code looks like this, and I never reflected on what that run arg actually was... 🤷♀
const drivers = {
    ChromeDriver,
    MediaDriver,
    RecorderDriver: makeRecorderDriver(api),
    WongDriver,
};

// tslint:disable-next-line:no-expression-statement
run(main, drivers);

@staltz I came to your issue from twitter.
May that fix it?
[P in keyof (So & Si)]

Replace by:
[P in (keyof So) & (keyof Si)]

Good fix! I think that did the trick. PR coming soon

I'm not sure the suggestion helped.
As far as I understand, keyof (So & Si) picks all the keys that exist in either So or Si. (keyof So) & (keyof Si) picks only keys of So that are also keys of Si. keyof T returns a union type of strings. And unionTypeOfStringsA & unionTypeOfStringsB is basically the intersection of those.

interface So {
  a: string
  b: string
}

interface Si {
  b: string
  c: string
}

So & Si  // interface {  a: string, b: string, c: string  }

keyof (So & Si) // 'a' | 'b' | 'c'

keyof So // 'a' | 'b'
keyof Si // 'b' | 'c'

(keyof So) & (keyof Si) // ('a' | 'b') & ('b' | 'c')   -->  'b'

Is the package with fixed typings going to be published any time soon? Just checked 4.4.0 and it doesn't have the changes yet

I finished this up, it's currently waiting for a review and release


Cypress CI failed with "detected that the Electron Renderer process just crashed"

What is the bug?
The local run can be passed but failed on CI environment
  Documents layer

We detected that the Electron Renderer process just crashed.

We have failed the current spec but will continue running the next spec.

This can happen for a number of different reasons.

If you're running lots of tests on a memory intense application.
  - Try increasing the CPU/memory on the machine you're running on.
  - Try enabling experimentalMemoryManagement in your config file.
  - Try lowering numTestsKeptInMemory in your config file during 'cypress open'.

You can learn more here:

https://on.cypress.io/renderer-process-crashed

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        2                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      1                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 minute, 18 seconds                                                             │
  │ Spec Ran:     documentsLayer.cy.js                                                             │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

How can one reproduce the bug?
Failed on CI, example: https://github.com/opensearch-project/dashboards-maps/actions/runs/8053561689/job/21996241052
What is the expected behavior?
Pass CI test

Catch All Triage - 1 2 3 4 5
@junqiu-lei Was this fixed?

Hi @dblock this CI issue most likely related to cypress version/github runner capacities, we are trying to replace this maps repo CI with test pipeline from opensearch-dashboards-functional-test which is stable without issue. The maps tests for 2.15 version on release pipeline CI currently is passed. Adding tag with 2.16 for this issue.


Improve node list layout

$ kontena node list
Name                                                                   Status     Labels
misty-morning-45                                                       online     region=eu-west-1,az=c
billowing-brook-52                                                     online     region=eu-west-1,az=b
hidden-river-37                                                        online     region=eu-west-1,az=a

$ kontena node list --all
Name                                                                   Status     Labels
development-grid/silent-rain-86                                        online     env=dev
production-grid/misty-morning-45                                       online     region=eu-west-1,az=c
production-grid/billowing-brook-52                                     online     region=eu-west-1,az=b
production-grid/hidden-river-37                                        online     region=eu-west-1,az=a

Fixes #473

why not make the columns dynamic in width? (eg like docker ps)

I think we should make them dynamic. But imho this new layout is cleaner and all the details can be still seen from node show command.
/cc @miskun @nevalla


Revert "fix(apps/prod/tekton/configs/triggers): fix git tag triggers"

Reverts PingCAP-QE/ee-ops#1071

/approve


shrink gcloud disk size

As the doxygen builds are no-longer being copied to this volume, the size was
rather excessive.

Could someone give this a quick look?  I suspect it would be better to edit  values in yaml files in place rather than completely overwriting them but feared this might end up being tiddly and taking more time then just creating new files from scratch.


add matching on string value having only alpha characters to matchCriteria

Add testing for a string value having only alpha characters to satisfy matchCriteria

Resolved in 1.4.273


Add "Entrepreneur" option to the list of Profession and cater for responsive list in smaller screen sizes

I believe there are often entrepreneurs or at least those with side businesses who attend the front-end coders meetup.
I propose adding the "Entrepreneur" option before the "Other" one, to the list of Professionals found in the https://frontend.mu/user/me URL, specifically in the RSVP details section.
A screenshot taken as of 16/02/2024:

@MrSunshyne recommends having it responsive on smaller screens.

Created PR https://github.com/Front-End-Coders-Mauritius/frontendmu-astro/pull/131


added-dropdown-menu

Fixes #91
onHover/onClick dropdown menu functionality added

@PhilippeduPreez one more thing: changeset 😀

You don't need to create a new variable, only to assign a value of another one to it ;)
https://github.com/flowershow/flowershow/blob/09e13c552ad254c34f5d37c1f850e1ca569380c7/templates/default/components/NavItem.js#L21


Using anonymous arrow function incorrectly shows 'possibly undefined value'

/* @flow */

function foo(bar?: Function) {
  return bar ? () => bar() : null;
}

gives
4:   return bar ? () => bar() : null;
                        ^ function call. Function cannot be called on possibly undefined value
4:   return bar ? () => bar() : null;
                        ^ undefined

https://flow.org/try/#0PQKgBAAgZgNg9gdzCYAoVUCuA7AxgFwEs5swo44AKAIwEMAnAfgC4wAxHA47ASjAG9UYMPQCm+TPVJ16YRmEp8AvAD4wMxWFbZMMGAG5UAX1RA

the problem is that flow is pessimistic about refinements. function arguments are let bindings (ie. you can reassign them), so inside the returned arrow function flow invalidates the refinement. obviously in this case we can see that nobody will reassign bar, but flow is not (yet) able to do this analysis. to solve this, you can use experimental.const_params to use const bindings in function arguments (ie. assigning to a parameter is never allowed), or you can use a placeholder variable:
function foo(_bar?: Function) {
  const bar = _bar;
  return bar ? () => bar() : null;
}

Error fixed and try flow no longer throws error.


why Use single quotes for strings & Extra semicolon ?

After you did this, the codes looks very ugly & not conform with the ECMAScript standard & JSON standard！

Then don't use it, totally an optional thing, no one is going to force you to use it.

ofcourse!

not conform with the ECMAScript standard

What do you mean with this? The code style that we are providing is 100% in line with the ECMAScript standard as far as I can understand...

JSON recommanded to using double quotes!
https://jsonformatter.curiousconcept.com/
{
    "key1":"value1",
    "key2":"value2",
    "key3":"value3"
}

{
     'key1': "value1",
    "key2": "value2",
    "key3": "value3"
}

{
    "key1": 'value1',
    "key2": "value2",
    "key3": "value3"
}

{
      key1: 'value1',
    "key2": "value2",
    "key3": "value3"
}

RFC 4627

https://www.ietf.org/rfc/rfc4627.txt
https://tools.ietf.org/html/rfc4627
2.5.  Strings

   The representation of strings is similar to conventions used in the C
   family of programming languages.  A string begins and ends with
   quotation marks.  All Unicode characters may be placed within the
   quotation marks except for the characters that must be escaped:
   quotation mark, reverse solidus, and the control characters (U+0000
   through U+001F).

   Any character may be escaped.  If the character is in the Basic
   Multilingual Plane (U+0000 through U+FFFF), then it may be
   represented as a six-character sequence: a reverse solidus, followed
   by the lowercase letter u, followed by four hexadecimal digits that
   encode the character's code point.  The hexadecimal letters A though
   F can be upper or lowercase.  So, for example, a string containing
   only a single reverse solidus character may be represented as
   "\u005C".

   Alternatively, there are two-character sequence escape
   representations of some popular characters.  So, for example, a
   string containing only a single reverse solidus character may be
   represented more compactly as "\\".

   To escape an extended character that is not in the Basic Multilingual
   Plane, the character is represented as a twelve-character sequence,
   encoding the UTF-16 surrogate pair.  So, for example, a string
   containing only the G clef character (U+1D11E) may be represented as
   "\uD834\uDD1E".

ECMA-404

ECMA-404 The JSON Data Interchange Standard.

http://www.json.org/
https://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf

That is JSON not JavaScript which has nothing to do with this Standard. This is a Javascript standard not a JSON standard.

As previous comment said, you are talking about JSON, not JavaScript...
I don't think that many people at all are always quoting their keys when they write javascript, and anyhow, we would be able to change that at this point...
Closing since I don't think that we need to change anything...


ROCm 6.1 Installers

I'm still working out some problems with the SUSE 15.5 installer.
When creating the docker image, the amdgpu-install is not generating the repo files correctly.
I can generate them myself, but then the rocm install is failing due to an unrecognised GPG key

@dgaliffiAMD PR #352 should fix some of the warnings about deprecated node16 packages in CI and potentially stabilize some of the failing tests


Tests fail on ppc64 with Ruby 1.9.3

Hi,
I'm having a problem with running tests on ppc64, running with Ruby 1.9.3, I'm getting this error:

Failure:
test_db_with_utf16(SQLite3::TestEncoding) [/builddir/build/BUILDROOT/stack_rails_3.0_rubygem-sqlite3-1.3.4-1.1.el6.ppc/opt/redhat/rails_3.0/root/usr/share/ruby/gems/1.9.1/gems/sqlite3-1.3.4/test/test_encoding.rb:79]:
<#Encoding:UTF-16LE> expected but was
<#Encoding:UTF-16BE>.

On 1.8.7, this problem doesn't occur, so it seems that this problem is connected with version of Ruby used. Any thoughts on how to fix it?
Thanks,
Bohuslav Kabrda

Please see issue #40

Sorry, but you have probably not noticed, that this failing tests is different one then the one reported by voxik in #40.

Still failing, but with different error:

Failure:
test_db_with_utf16(SQLite3::TestEncoding) [/root/rpmbuild/BUILD/rubygem-sqlite3-1.3.4/usr/share/ruby/gems/1.9.1/gems/sqlite3-1.3.4/test/test_encoding.rb:81]:
<#Encoding:UTF-16BE> expected but was
<#Encoding:UTF-8>.

Would you mind creating a patch ?

I'd be glad, but I'm not sure about the right behaviour. Are these outputs ok and only the tests are wrong or is the library itself doing something bad? I think it's the first case, but I'd like to make this clear.

The issue appear to be endian differences. Seems is now taking the string
encoding instead. Will check when arrive to the office .
On Oct 21, 2011 9:15 AM, "bkabrda" <
<EMAIL_ADDRESS>wrote:

I'd be glad, but I'm not sure about the right behaviour. Are these outputs
ok and only the tests are wrong or is the library itself doing something
bad? I think it's the first case, but I'd like to make this clear.
--
Reply to this email directly or view it on GitHub:
https://github.com/luislavena/sqlite3-ruby/issues/51#issuecomment-2478419

I don't have access to a PPC platform so I'm guessing, can you try out the following in IRB?
>> utf16 = ([1].pack("I") == [1].pack("N")) ? "UTF-16BE" : "UTF-16LE"
=> "UTF-16LE"
>> a = ":memory:".encode(utf16)
=> ":memory:"
>> a.encoding
=> #<Encoding:UTF-16LE>
>> Encoding.find(utf16)
=> #<Encoding:UTF-16LE>
>> a.encoding == Encoding.find(utf16)
=> true

utf16 = ([1].pack("I") == [1].pack("N")) ? "UTF-16BE" : "UTF-16LE"
=> "UTF-16BE"
a = ":memory:".encode(utf16)
=> ":memory:"
a.encoding
=> #Encoding:UTF-16BE
Encoding.find(utf16)
=> #Encoding:UTF-16BE
a.encoding == Encoding.find(utf16)
=> true

Perfect, thank you. Now, can you cd into sqlite3, and after compilation, do this:
$ irb -Ilib
>> utf16 = ([1].pack("I") == [1].pack("N")) ? "UTF-16BE" : "UTF-16LE"
=> "UTF-16LE"
>> require "sqlite3"
=> true
>> db = SQLite3::Database.new(':memory:'.encode(utf16))
=> #<SQLite3::Database:0x00000100a04dc0>
>> db.encoding
=> #<Encoding:UTF-16LE>
>> db.close
=> #<SQLite3::Database:0x00000100a04dc0>
>> exit

Here it is:

>> utf16 = ([1].pack("I") == [1].pack("N")) ? "UTF-16BE" : "UTF-16
=> "UTF-16BE"
>> require "sqlite3"
=> true
>> db = SQLite3::Database.new(':memory:'.encode(utf16))
=> #<SQLite3::Database:0x0001003e7dca78>
>> db.encoding
=> #<Encoding:UTF-8>
>> db.close
=> #<SQLite3::Database:0x0001003e7dca78>

Ok, the problem is here:
https://github.com/luislavena/sqlite3-ruby/blob/master/ext/sqlite3/database.c#L56
It only considers 16LE and not 16BE.
@tenderlove, do you remember if SQLite3 supports it in it's open16 function?

Can you try applying the following patch?

diff --git a/ext/sqlite3/database.c b/ext/sqlite3/database.c
index 2a1005a..4857c55 100644
--- a/ext/sqlite3/database.c
+++ b/ext/sqlite3/database.c
@@ -53,7 +53,7 @@ static VALUE initialize(int argc, VALUE *argv, VALUE self)
   if(NIL_P(opts)) opts = rb_hash_new();

 #ifdef HAVE_RUBY_ENCODING_H
-  if(UTF16_LE_P(file)) {
+  if(UTF16_LE_P(file) || UTF16_BE_P(file)) {
     status = sqlite3_open16(utf16_string_value_ptr(file), &ctx->db);
   } else {
 #endif
diff --git a/ext/sqlite3/sqlite3_ruby.h b/ext/sqlite3/sqlite3_ruby.h
index 3318ac6..ec70b9c 100644
--- a/ext/sqlite3/sqlite3_ruby.h
+++ b/ext/sqlite3/sqlite3_ruby.h
@@ -21,6 +21,7 @@

 #define UTF8_P(_obj) (rb_enc_get_index(_obj) == rb_utf8_encindex())
 #define UTF16_LE_P(_obj) (rb_enc_get_index(_obj) == rb_enc_find_index("UTF-16LE"))
+#define UTF16_BE_P(_obj) (rb_enc_get_index(_obj) == rb_enc_find_index("UTF-16BE"))
 #define SQLITE3_UTF8_STR_NEW2(_obj) \
     (rb_enc_associate_index(rb_str_new2(_obj), rb_utf8_encindex()))

Sure, thanks a lot. I will try it on Monday when I get to work and I'll let you know.

Well, the tests keep failing, the output is the same as in #issuecomment-2479430. I did some debugging and I think I found where the error is - the condition
if(UTF16_LE_P(file) || UTF16_BE_P(file)) {
doesn't evaluate to true (and thus the program goes into the else section). This is particularly confusing, as trying to evaluate the condition by myself I got:
(gdb) call UTF16_LE_P(file)
$65 = 0
(gdb) call UTF16_BE_P(file)
$66 = 1
(gdb) call UTF16_BE_P(file) || UTF16_BE_P(file)
$67 = 1
So I got true, but the program always goes to else section.

Well, from what I can see, still calling sqlite3_open16 on my little endian system with UTF-16BE generates a UTF-16LE encoding for the database, so I guess will need access to an big endian system to test this out.
I can't do anything else to help you so I'm going to wait if you find what is going on. Perhaps is the version of SQLite3 you're building against, can't say.
Other than that, I'm out of ideas.

The problem is, that the condition doesn't get evaluated as true although it should. And therefore sqlite3_open16 doesn't get called at all.

Mistake is on my side, I'm so sorry :) Your patch works great. Thanks.


How can I style the search bar ?

Can you provide me with an example of styling the search bar because when try to use style={{ backroundColor: '#FAFAFA' }} then, search bar disappears.

So it seems like you cant. I did it by modifying the source myself.
You need 'barTintColor' which you can define just as 'tintColor' has been defined in 3 files:
https://github.com/umhan35/react-native-search-bar/search?utf8=✓&q=tintColor&type=Code

I did solve this same problem by adding customStyles prop to search bar.
So on search bar render function i added new prop on style attribute. On customStyles prop i have {backgroundColor: '#ffffff'}. What do you think about this solution? I think this gives quite much more freedom on developing.
<RNSearchBar       style={[{height: NativeModules.RNSearchBarManager.ComponentHeight}, this.props.customStyles]}       onChange={this._onChange}       onPress={this._onPress}       {...this.props}     />;

If you set hideBackground=true, and wrap the SearchBar with a View, you can style the background anyway you like


Touchup Get-DbaWaitStatistics

Minor adjustments to help text and format.
No change to logic.

thanks for the approval!


zero_optimization.cpu_offload: true leads to a silent crash

I'm experimenting with various zero_optimization config options and I noticed that when I flip to true zero_optimization.cpu_offload, the application exits w/o crashing or doing any training.
{
    "train_batch_size": 20,
    "steps_per_print": 2000,

    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

   "zero_optimization": {
       "stage": 0,
       "allgather_partitions": true,
       "allgather_bucket_size": 500000000,
       "overlap_comm": true,
       "reduce_scatter": true,
       "reduce_bucket_size": 500000000,
       "contiguous_gradients": false,
       "cpu_offload": false
   },

   "optimizer": {
     "type": "Adam",
     "params": {
       "lr": 3e-5,
       "betas": [
         0.8,
         0.999
       ],
       "eps": 1e-8,
       "weight_decay": 3e-7
     }
   },
   "scheduler": {
     "type": "WarmupLR",
     "params": {
       "warmup_min_lr": 0,
       "warmup_max_lr": 3e-5,
       "warmup_num_steps": 500
     }
   },
   "wall_clock_breakdown": false
}

leads to a silent exit but doing nothing:

Full log

export BS=20; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0 deepspeed  ./finetune_trainer.py --model_name_or_path sshleifer/distill-mbart-en-ro-12-4 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500 --deepspeed --deepspeed_config ds_config.json
rm: cannot remove 'output_dir': No such file or directory
[2020-12-18 19:42:37,871] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2020-12-18 19:42:37,897] [INFO] [runner.py:355:main] cmd = /home/stas/anaconda3/envs/main-38/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=<IP_ADDRESS> --master_port=29500 ./finetune_trainer.py --model_name_or_path sshleifer/distill-mbart-en-ro-12-4 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size 20 --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500 --deepspeed --deepspeed_config ds_config.json
[2020-12-18 19:42:38,631] [INFO] [launch.py:78:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2020-12-18 19:42:38,631] [INFO] [launch.py:84:main] nnodes=1, num_local_procs=2, node_rank=0
[2020-12-18 19:42:38,631] [INFO] [launch.py:99:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2020-12-18 19:42:38,631] [INFO] [launch.py:100:main] dist_world_size=2
[2020-12-18 19:42:38,631] [INFO] [launch.py:102:main] Setting CUDA_VISIBLE_DEVICES=0,1
['--deepspeed', '--deepspeed_config', 'ds_config.json']
1
2020-12-18 19:42:40 | WARNING | main | Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, 16-bits training: False
2020-12-18 19:42:40 | INFO | main | Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='output_dir', overwrite_output_dir=True, do_train=True, do_eval=False, do_predict=False, model_parallel=False, evaluation_strategy=<EvaluationStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=20, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=3e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-06, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, warmup_steps=500, logging_dir='runs/Dec18_19-42-40_hope', logging_first_step=True, logging_steps=1000, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, dataloader_num_workers=0, past_index=-1, run_name='output_dir', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, fp16_backend='auto', sharded_ddp=False, label_smoothing=0.1, sortish_sampler=True, predict_with_generate=False, adafactor=False, encoder_layerdrop=None, decoder_layerdrop=None, dropout=None, attention_dropout=None, lr_scheduler='linear')
['--deepspeed', '--deepspeed_config', 'ds_config.json']
0
2020-12-18 19:42:40 | WARNING | main | Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, 16-bits training: False
2020-12-18 19:42:40 | INFO | main | Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='output_dir', overwrite_output_dir=True, do_train=True, do_eval=False, do_predict=False, model_parallel=False, evaluation_strategy=<EvaluationStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=20, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=3e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-06, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, warmup_steps=500, logging_dir='runs/Dec18_19-42-40_hope', logging_first_step=True, logging_steps=1000, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=0, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, dataloader_num_workers=0, past_index=-1, run_name='output_dir', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, fp16_backend='auto', sharded_ddp=False, label_smoothing=0.1, sortish_sampler=True, predict_with_generate=False, adafactor=False, encoder_layerdrop=None, decoder_layerdrop=None, dropout=None, attention_dropout=None, lr_scheduler='linear')
[INFO|configuration_utils.py:431] 2020-12-18 19:42:41,139 >> loading configuration file https://huggingface.co/sshleifer/distill-mbart-en-ro-12-4/resolve/main/config.json from cache at /home/stas/.cache/huggingface/transformers/3a05b98cd4a37d1704b3d884e5bd1e19a3783d2d0a9f1f5449f4896f4d163781.b57423f4136691c59b9844b9358d5b26655ad2a5e080f0fbb24070bc528d090e
[INFO|configuration_utils.py:467] 2020-12-18 19:42:41,141 >> Model config MBartConfig {
"_num_labels": 3,
"activation_dropout": 0.0,
"activation_function": "gelu",
"add_bias_logits": false,
"add_final_layer_norm": true,
"architectures": [
"BartForConditionalGeneration"
],
"attention_dropout": 0.0,
"bos_token_id": 0,
"classif_dropout": 0.0,
"classifier_dropout": 0.0,
"d_model": 1024,
"decoder_attention_heads": 16,
"decoder_ffn_dim": 4096,
"decoder_layerdrop": 0.0,
"decoder_layers": 4,
"decoder_start_token_id": 250020,
"do_blenderbot_90_layernorm": false,
"dropout": 0.1,
"encoder_attention_heads": 16,
"encoder_ffn_dim": 4096,
"encoder_layerdrop": 0.0,
"encoder_layers": 12,
"eos_token_id": 2,
"extra_pos_embeddings": 2,
"force_bos_token_to_be_generated": false,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1",
"2": "LABEL_2"
},
"init_std": 0.02,
"is_encoder_decoder": true,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1,
"LABEL_2": 2
},
"max_length": 1000,
"max_position_embeddings": 1024,
"model_type": "mbart",
"normalize_before": true,
"normalize_embedding": true,
"num_beams": 5,
"num_hidden_layers": 12,
"output_past": true,
"pad_token_id": 1,
"save_step": 7,
"scale_embedding": true,
"static_position_embeddings": false,
"use_cache": true,
"variant": "prelayernorm",
"vocab_size": 250027
}
[INFO|configuration_utils.py:431] 2020-12-18 19:42:41,415 >> loading configuration file https://huggingface.co/sshleifer/distill-mbart-en-ro-12-4/resolve/main/config.json from cache at /home/stas/.cache/huggingface/transformers/3a05b98cd4a37d1704b3d884e5bd1e19a3783d2d0a9f1f5449f4896f4d163781.b57423f4136691c59b9844b9358d5b26655ad2a5e080f0fbb24070bc528d090e
[INFO|configuration_utils.py:467] 2020-12-18 19:42:41,417 >> Model config MBartConfig {
"_num_labels": 3,
"activation_dropout": 0.0,
"activation_function": "gelu",
"add_bias_logits": false,
"add_final_layer_norm": true,
"architectures": [
"BartForConditionalGeneration"
],
"attention_dropout": 0.0,
"bos_token_id": 0,
"classif_dropout": 0.0,
"classifier_dropout": 0.0,
"d_model": 1024,
"decoder_attention_heads": 16,
"decoder_ffn_dim": 4096,
"decoder_layerdrop": 0.0,
"decoder_layers": 4,
"decoder_start_token_id": 250020,
"do_blenderbot_90_layernorm": false,
"dropout": 0.1,
"encoder_attention_heads": 16,
"encoder_ffn_dim": 4096,
"encoder_layerdrop": 0.0,
"encoder_layers": 12,
"eos_token_id": 2,
"extra_pos_embeddings": 2,
"force_bos_token_to_be_generated": false,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1",
"2": "LABEL_2"
},
"init_std": 0.02,
"is_encoder_decoder": true,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1,
"LABEL_2": 2
},
"max_length": 1000,
"max_position_embeddings": 1024,
"model_type": "mbart",
"normalize_before": true,
"normalize_embedding": true,
"num_beams": 5,
"num_hidden_layers": 12,
"output_past": true,
"pad_token_id": 1,
"save_step": 7,
"scale_embedding": true,
"static_position_embeddings": false,
"use_cache": true,
"variant": "prelayernorm",
"vocab_size": 250027
}
[INFO|tokenization_utils_base.py:1718] 2020-12-18 19:42:41,418 >> Model name 'sshleifer/distill-mbart-en-ro-12-4' not found in model shortcut name list (facebook/mbart-large-en-ro, facebook/mbart-large-cc25). Assuming 'sshleifer/distill-mbart-en-ro-12-4' is a path, a model identifier, or url to a directory containing tokenizer files.
[INFO|tokenization_utils_base.py:1802] 2020-12-18 19:42:42,925 >> loading file https://huggingface.co/sshleifer/distill-mbart-en-ro-12-4/resolve/main/sentencepiece.bpe.model from cache at /home/stas/.cache/huggingface/transformers/62ed1799c9b9a3c199222637281d38762ae87e00165a2613e31c93b3673f08b8.00628a9eeb8baf4080d44a0abe9fe8057893de20c7cb6e6423cddbf452f7d4d8
[INFO|tokenization_utils_base.py:1802] 2020-12-18 19:42:42,925 >> loading file https://huggingface.co/sshleifer/distill-mbart-en-ro-12-4/resolve/main/added_tokens.json from cache at None
[INFO|tokenization_utils_base.py:1802] 2020-12-18 19:42:42,925 >> loading file https://huggingface.co/sshleifer/distill-mbart-en-ro-12-4/resolve/main/special_tokens_map.json from cache at /home/stas/.cache/huggingface/transformers/9423d956f3dd4d8fd97112a8d3f87081f6256ce54ccfecd27938c48e294b8aa8.72fa8565f9c8b5dc27e7ac070020aec80359d9da2e5628b3f313f41bf44d322c
[INFO|tokenization_utils_base.py:1802] 2020-12-18 19:42:42,925 >> loading file https://huggingface.co/sshleifer/distill-mbart-en-ro-12-4/resolve/main/tokenizer_config.json from cache at /home/stas/.cache/huggingface/transformers/f5629ec54e86b66e2e9879777df84ce24ede4c93495e6ce9f9161011260c5344.67d01b18f2079bd75eac0b2f2e7235768c7f26bd728e7a855a1c5acae01a91a8
[INFO|tokenization_utils_base.py:1802] 2020-12-18 19:42:42,925 >> loading file https://huggingface.co/sshleifer/distill-mbart-en-ro-12-4/resolve/main/tokenizer.json from cache at None
[INFO|tokenization_utils_base.py:925] 2020-12-18 19:42:43,989 >> Assigning ['ar_AR', 'cs_CZ', 'de_DE', 'en_XX', 'es_XX', 'et_EE', 'fi_FI', 'fr_XX', 'gu_IN', 'hi_IN', 'it_IT', 'ja_XX', 'kk_KZ', 'ko_KR', 'lt_LT', 'lv_LV', 'my_MM', 'ne_NP', 'nl_XX', 'ro_RO', 'ru_RU', 'si_LK', 'tr_TR', 'vi_VN', 'zh_CN'] to the additional_special_tokens key of the tokenizer
[INFO|modeling_utils.py:1024] 2020-12-18 19:42:44,314 >> loading weights file https://huggingface.co/sshleifer/distill-mbart-en-ro-12-4/resolve/main/pytorch_model.bin from cache at /home/stas/.cache/huggingface/transformers/d2a7ade93d629fb16e06233407ab8aa0e70af5532c66c3b38ce2ff905743bf78.fa8ebf3af9c5dec8982ce624e74de87e85c9a944e776b79b8e8bd65126ed2073
Some weights of MBartForConditionalGeneration were not initialized from the model checkpoint at sshleifer/distill-mbart-en-ro-12-4 and are newly initialized: ['lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|modeling_utils.py:1045] 2020-12-18 19:43:06,939 >> load time=0.8602
[2020-12-18 19:43:07,280] [INFO] [logging.py:60:log_dist] [Rank -1] DeepSpeed info: version=0.3.8+fd2f970, git-hash=fd2f970, git-branch=master
[2020-12-18 19:43:07,280] [INFO] [engine.py:147:init] Initializing torch distributed with backend: nccl
[INFO|modeling_utils.py:1145] 2020-12-18 19:43:07,318 >> All model checkpoint weights were used when initializing MBartForConditionalGeneration.
[WARNING|modeling_utils.py:1147] 2020-12-18 19:43:07,318 >> Some weights of MBartForConditionalGeneration were not initialized from the model checkpoint at sshleifer/distill-mbart-en-ro-12-4 and are newly initialized: ['lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[2020-12-18 19:43:07,512] [INFO] [logging.py:60:log_dist] [Rank -1] DeepSpeed info: version=0.3.8+fd2f970, git-hash=fd2f970, git-branch=master
[2020-12-18 19:43:07,512] [INFO] [engine.py:147:init] Initializing torch distributed with backend: nccl
[2020-12-18 19:43:11,225] [INFO] [engine.py:70:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
[2020-12-18 19:43:11,229] [INFO] [engine.py:70:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
Adam Optimizer #0 is created with AVX2 arithmetic capability.
Config: alpha=0.000030, betas=(0.800000, 0.999000), weight_decay=0.000000, adam_w=1
[2020-12-18 19:43:13,258] [INFO] [engine.py:702:_configure_fp16_optimizer] Creating fp16 unfused optimizer with dynamic loss scale
Adam Optimizer #0 is created with AVX2 arithmetic capability.
Config: alpha=0.000030, betas=(0.800000, 0.999000), weight_decay=0.000000, adam_w=1
[2020-12-18 19:43:13,262] [INFO] [engine.py:593:_configure_optimizer] Using DeepSpeed Optimizer param name adam as basic optimizer
[2020-12-18 19:43:13,262] [INFO] [engine.py:598:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam (
Parameter Group 0
amsgrad: False
betas: [0.8, 0.999]
bias_correction: True
eps: 1e-08
lr: 3e-05
weight_decay: 3e-07
)
[2020-12-18 19:43:13,262] [INFO] [engine.py:702:_configure_fp16_optimizer] Creating fp16 unfused optimizer with dynamic loss scale
[2020-12-18 19:43:13,262] [INFO] [unfused_optimizer.py:36:init] Fused Lamb Legacy : False
group 0 param 0 = 1048576
group 0 param 0 = 1048576

If I flip zero_optimization.cpu_offload to false everything works:

Full log

export BS=20; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0 deepspeed  ./finetune_trainer.py --model_name_or_path sshleifer/tiny-mbart --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500 --deepspeed --deepspeed_config ds_config.json
rm: cannot remove 'output_dir': No such file or directory
[2020-12-18 20:29:55,608] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2020-12-18 20:29:55,634] [INFO] [runner.py:355:main] cmd = /home/stas/anaconda3/envs/main-38/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=<IP_ADDRESS> --master_port=29500 ./finetune_trainer.py --model_name_or_path sshleifer/tiny-mbart --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size 20 --sortish_sampler --src_lang en_XX --task translation --tgt_lang ro_RO --val_max_target_length 128 --warmup_steps 500 --n_train 500 --deepspeed --deepspeed_config ds_config.json
[2020-12-18 20:29:56,371] [INFO] [launch.py:78:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2020-12-18 20:29:56,372] [INFO] [launch.py:84:main] nnodes=1, num_local_procs=2, node_rank=0
[2020-12-18 20:29:56,372] [INFO] [launch.py:99:main] global_rank_mapping=defaultdict(, {'localhost': [0, 1]})
[2020-12-18 20:29:56,372] [INFO] [launch.py:100:main] dist_world_size=2
[2020-12-18 20:29:56,372] [INFO] [launch.py:102:main] Setting CUDA_VISIBLE_DEVICES=0,1
['--deepspeed', '--deepspeed_config', 'ds_config.json']
1
2020-12-18 20:29:58 | WARNING | __main__ | Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, 16-bits training: False
2020-12-18 20:29:58 | INFO | __main__ | Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='output_dir', overwrite_output_dir=True, do_train=True, do_eval=False, do_predict=False, model_parallel=False, evaluation_strategy=, prediction_loss_only=False, per_device_train_batch_size=20, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=3e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-06, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, warmup_steps=500, logging_dir='runs/Dec18_20-29-58_hope', logging_first_step=True, logging_steps=1000, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, dataloader_num_workers=0, past_index=-1, run_name='output_dir', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, fp16_backend='auto', sharded_ddp=False, label_smoothing=0.1, sortish_sampler=True, predict_with_generate=False, adafactor=False, encoder_layerdrop=None, decoder_layerdrop=None, dropout=None, attention_dropout=None, lr_scheduler='linear')
['--deepspeed', '--deepspeed_config', 'ds_config.json']
0
2020-12-18 20:29:58 | WARNING | __main__ | Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, 16-bits training: False
2020-12-18 20:29:58 | INFO | __main__ | Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='output_dir', overwrite_output_dir=True, do_train=True, do_eval=False, do_predict=False, model_parallel=False, evaluation_strategy=, prediction_loss_only=False, per_device_train_batch_size=20, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=3e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-06, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, warmup_steps=500, logging_dir='runs/Dec18_20-29-58_hope', logging_first_step=True, logging_steps=1000, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=0, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, dataloader_num_workers=0, past_index=-1, run_name='output_dir', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, fp16_backend='auto', sharded_ddp=False, label_smoothing=0.1, sortish_sampler=True, predict_with_generate=False, adafactor=False, encoder_layerdrop=None, decoder_layerdrop=None, dropout=None, attention_dropout=None, lr_scheduler='linear')
[INFO|configuration_utils.py:431] 2020-12-18 20:29:58,890 >> loading configuration file https://huggingface.co/sshleifer/tiny-mbart/resolve/main/config.json from cache at /home/stas/.cache/huggingface/transformers/5fd8333015b256440e1b6fbf2d5f86a4868a39440a89554475ee8d1c616d9e56.5b830f48cd63bb457b6ea960d512d839da5b4c30ee8b6998c04977316c32b2f0
[INFO|configuration_utils.py:467] 2020-12-18 20:29:58,892 >> Model config MBartConfig {
  "_num_labels": 3,
  "activation_dropout": 0.0,
  "activation_function": "gelu",
  "add_bias_logits": false,
  "add_final_layer_norm": true,
  "architectures": [
    "BartForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "classifier_dropout": 0.0,
  "d_model": 2,
  "decoder_attention_heads": 1,
  "decoder_ffn_dim": 4,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 2,
  "do_blenderbot_90_layernorm": false,
  "dropout": 0.1,
  "encoder_attention_heads": 1,
  "encoder_ffn_dim": 4,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 2,
  "eos_token_id": 2,
  "extra_pos_embeddings": 2,
  "force_bos_token_to_be_generated": false,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "max_position_embeddings": 1024,
  "model_type": "mbart",
  "normalize_before": true,
  "normalize_embedding": true,
  "num_beams": 2,
  "num_hidden_layers": 2,
  "output_past": true,
  "pad_token_id": 1,
  "scale_embedding": true,
  "static_position_embeddings": false,
  "use_cache": true,
  "vocab_size": 250027
}
[INFO|configuration_utils.py:431] 2020-12-18 20:29:59,191 >> loading configuration file https://huggingface.co/sshleifer/tiny-mbart/resolve/main/config.json from cache at /home/stas/.cache/huggingface/transformers/5fd8333015b256440e1b6fbf2d5f86a4868a39440a89554475ee8d1c616d9e56.5b830f48cd63bb457b6ea960d512d839da5b4c30ee8b6998c04977316c32b2f0
[INFO|configuration_utils.py:467] 2020-12-18 20:29:59,192 >> Model config MBartConfig {
"_num_labels": 3,
"activation_dropout": 0.0,
"activation_function": "gelu",
"add_bias_logits": false,
"add_final_layer_norm": true,
"architectures": [
"BartForConditionalGeneration"
],
"attention_dropout": 0.0,
"bos_token_id": 0,
"classif_dropout": 0.0,
"classifier_dropout": 0.0,
"d_model": 2,
"decoder_attention_heads": 1,
"decoder_ffn_dim": 4,
"decoder_layerdrop": 0.0,
"decoder_layers": 2,
"do_blenderbot_90_layernorm": false,
"dropout": 0.1,
"encoder_attention_heads": 1,
"encoder_ffn_dim": 4,
"encoder_layerdrop": 0.0,
"encoder_layers": 2,
"eos_token_id": 2,
"extra_pos_embeddings": 2,
"force_bos_token_to_be_generated": false,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1",
"2": "LABEL_2"
},
"init_std": 0.02,
"is_encoder_decoder": true,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1,
"LABEL_2": 2
},
"max_position_embeddings": 1024,
"model_type": "mbart",
"normalize_before": true,
"normalize_embedding": true,
"num_beams": 2,
"num_hidden_layers": 2,
"output_past": true,
"pad_token_id": 1,
"scale_embedding": true,
"static_position_embeddings": false,
"use_cache": true,
"vocab_size": 250027
}
[INFO|tokenization_utils_base.py:1718] 2020-12-18 20:29:59,192 >> Model name 'sshleifer/tiny-mbart' not found in model shortcut name list (facebook/mbart-large-en-ro, facebook/mbart-large-cc25). Assuming 'sshleifer/tiny-mbart' is a path, a model identifier, or url to a directory containing tokenizer files.
[INFO|tokenization_utils_base.py:1802] 2020-12-18 20:30:00,718 >> loading file https://huggingface.co/sshleifer/tiny-mbart/resolve/main/sentencepiece.bpe.model from cache at /home/stas/.cache/huggingface/transformers/13a2c62c1dabc5357bc38b0694f5829f3db0708d51f1a0f07734f62cc0a825a0.00628a9eeb8baf4080d44a0abe9fe8057893de20c7cb6e6423cddbf452f7d4d8
[INFO|tokenization_utils_base.py:1802] 2020-12-18 20:30:00,718 >> loading file https://huggingface.co/sshleifer/tiny-mbart/resolve/main/added_tokens.json from cache at None
[INFO|tokenization_utils_base.py:1802] 2020-12-18 20:30:00,718 >> loading file https://huggingface.co/sshleifer/tiny-mbart/resolve/main/special_tokens_map.json from cache at /home/stas/.cache/huggingface/transformers/33fa7894ab257a74cede3060dca6d2fc609918785e80160f6c057723ece47292.0dc5b1041f62041ebbd23b1297f2f573769d5c97d8b7c28180ec86b8f6185aa8
[INFO|tokenization_utils_base.py:1802] 2020-12-18 20:30:00,718 >> loading file https://huggingface.co/sshleifer/tiny-mbart/resolve/main/tokenizer_config.json from cache at /home/stas/.cache/huggingface/transformers/e9c580e6446c42ed20fb148206f2a9bd75a825278ffa029df063682077d45bb6.67d01b18f2079bd75eac0b2f2e7235768c7f26bd728e7a855a1c5acae01a91a8
[INFO|tokenization_utils_base.py:1802] 2020-12-18 20:30:00,718 >> loading file https://huggingface.co/sshleifer/tiny-mbart/resolve/main/tokenizer.json from cache at None
[INFO|tokenization_utils_base.py:925] 2020-12-18 20:30:01,779 >> Assigning ['ar_AR', 'cs_CZ', 'de_DE', 'en_XX', 'es_XX', 'et_EE', 'fi_FI', 'fr_XX', 'gu_IN', 'hi_IN', 'it_IT', 'ja_XX', 'kk_KZ', 'ko_KR', 'lt_LT', 'lv_LV', 'my_MM', 'ne_NP', 'nl_XX', 'ro_RO', 'ru_RU', 'si_LK', 'tr_TR', 'vi_VN', 'zh_CN'] to the additional_special_tokens key of the tokenizer
Some weights of MBartForConditionalGeneration were not initialized from the model checkpoint at sshleifer/tiny-mbart and are newly initialized: ['lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|modeling_utils.py:1024] 2020-12-18 20:30:02,107 >> loading weights file https://huggingface.co/sshleifer/tiny-mbart/resolve/main/pytorch_model.bin from cache at /home/stas/.cache/huggingface/transformers/d6eec704737db03a21a794f08b07fcbb71d855562a992cfb1be6193b37a7ff68.61ce63751e40ea882dd1a22b6c9303b954b81ec69d631ab0541750fd856720be
[INFO|modeling_utils.py:1045] 2020-12-18 20:30:02,150 >> load time=0.0017
[INFO|modeling_utils.py:1145] 2020-12-18 20:30:02,152 >> All model checkpoint weights were used when initializing MBartForConditionalGeneration.
 False
[2020-12-18 20:30:05,681] [INFO] [config.py:648:print]   amp_enabled .................. False
[2020-12-18 20:30:05,681] [INFO] [config.py:648:print]   amp_params ................... False
[2020-12-18 20:30:05,681] [INFO] [config.py:648:print]   disable_allgather ............ False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   dump_state ................... False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   dynamic_loss_scale_args ...... {'init_scale':<PHONE_NUMBER>, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   fp16_enabled ................. True
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   global_rank .................. 0
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   gradient_accumulation_steps .. 1
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   gradient_clipping ............ 0.0
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   gradient_predivide_factor .... 1.0
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   initial_dynamic_scale ........<PHONE_NUMBER>
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   loss_scale ................... 0
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   memory_breakdown ............. False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   optimizer_legacy_fusion ...... False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   optimizer_name ............... adam
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   optimizer_params ............. {'lr': 3e-05, 'betas': [0.8, 0.999], 'eps': 1e-08, 'weight_decay': 3e-07, 'adam_w_mode': True}
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   pld_enabled .................. False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   pld_params ................... False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   prescale_gradients ........... False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   scheduler_name ............... WarmupLR
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 3e-05, 'warmup_num_steps': 500}
2020-12-18 20:30:05 | INFO | main | *** Train ***
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   sparse_attention ............. None
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   sparse_gradients_enabled ..... False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   steps_per_print .............. 2000
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   tensorboard_enabled .......... False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   tensorboard_job_name ......... DeepSpeedJobName
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   tensorboard_output_path ......
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   train_batch_size ............. 20
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   train_micro_batch_size_per_gpu  10
2020-12-18 20:30:05 | WARNING | seq2seq_trainer | scheduler is passed to Seq2SeqTrainer, --lr_scheduler arg is ignored.
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   wall_clock_breakdown ......... False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   world_size ................... 2
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   zero_allow_untested_optimizer  False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   zero_config .................. {
"allgather_bucket_size": 500000000,
"allgather_partitions": true,
"contiguous_gradients": true,
"cpu_offload": false,
"elastic_checkpoint": true,
"load_from_fp32_weights": true,
"overlap_comm": false,
"reduce_bucket_size": 500000000,
"reduce_scatter": false,
"stage": 0
}
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   zero_enabled ................. False
[2020-12-18 20:30:05,682] [INFO] [config.py:648:print]   zero_optimization_stage ...... 0
[2020-12-18 20:30:05,682] [INFO] [config.py:650:print]   json = {
"fp16":{
"enabled":true,
"hysteresis":2,
"loss_scale":0,
"loss_scale_window":1000,
"min_loss_scale":1
},
"optimizer":{
"params":{
"adam_w_mode":true,
"betas":[
0.8,
0.999
],
"eps":1e-08,
"lr":3e-05,
"weight_decay":3e-07
},
"type":"Adam"
},
"scheduler":{
"params":{
"warmup_max_lr":3e-05,
"warmup_min_lr":0,
"warmup_num_steps":500
},
"type":"WarmupLR"
},
"steps_per_print":2000,
"train_batch_size":20,
"wall_clock_breakdown":false,
"zero_optimization":{
"allgather_bucket_size":500000000,
"allgather_partitions":true,
"contiguous_gradients":true,
"cpu_offload":false,
"overlap_comm":false,
"reduce_bucket_size":500000000,
"reduce_scatter":false,
"stage":0
}
}
FusedAdam (
Parameter Group 0
betas: [0.8, 0.999]
bias_correction: True
eps: 1e-08
lr: 3e-05
step: 1
weight_decay: 3e-07
)
<deepspeed.runtime.lr_schedules.WarmupLR object at 0x7f303160d640>
2020-12-18 20:30:05 | INFO | main | *** Train ***
2020-12-18 20:30:05 | WARNING | seq2seq_trainer | scheduler is passed to Seq2SeqTrainer, --lr_scheduler arg is ignored.
[INFO|trainer.py:723] 2020-12-18 20:30:05,688 >> ***** Running training *****
[INFO|trainer.py:724] 2020-12-18 20:30:05,688 >>   Num examples = 500
[INFO|trainer.py:725] 2020-12-18 20:30:05,688 >>   Num Epochs = 1
[INFO|trainer.py:726] 2020-12-18 20:30:05,688 >>   Instantaneous batch size per device = 20
[INFO|trainer.py:727] 2020-12-18 20:30:05,688 >>   Total train batch size (w. parallel, distributed & accumulation) = 40
[INFO|trainer.py:728] 2020-12-18 20:30:05,688 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:729] 2020-12-18 20:30:05,688 >>   Total optimization steps = 13
{'loss': inf, 'learning_rate': 0.0, 'epoch': 0.07692307692307693}
92%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍         | 12/13 [00:02<00:00,  5.65it/s][INFO|trainer.py:883] 2020-12-18 20:30:08,588 >>
Training completed. Do not forget to share your model on huggingface.co/models =)
{'epoch': 1.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:02<00:00,  5.95it/s]
[INFO|trainer.py:1247] 2020-12-18 20:30:08,589 >> Saving model checkpoint to output_dir
[INFO|trainer.py:1251] 2020-12-18 20:30:08,589 >> Trainer.model is not a PreTrainedModel, only saving its state dict.
2020-12-18 20:30:08 | INFO | main | ***** train metrics *****
2020-12-18 20:30:08 | INFO | main |   train_samples_per_second = 172.096
2020-12-18 20:30:08 | INFO | main |   train_runtime = 2.9054
2020-12-18 20:30:08 | INFO | main |   train_n_ojbs = 500

I know I haven't provided reproduction info, as I haven't quite finished working on integration with HF transformers, but it should be ready soon. I was hoping you could tell from logs what went wrong. But if it isn't helpful I will update this Issue with reproduction details once I have a transformers branch you could experiment with.

@stas00, thanks for reporting this issue. Can you please set zero_optimization.stage == 2? This is a requirement for zero_optimization.cpu_offload?

Thank you, @tjruwase
Would you add an assert to help the user to know that? Surely silent exit w/o a traceback must be incomplete, right?
When I did that - it moves a long and then crashes:
Traceback (most recent call last):
  File "./finetune_trainer.py", line 352, in <module>
    main()
  File "./finetune_trainer.py", line 289, in main
    train_result = trainer.train(
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/src/transformers/trainer.py", line 825, in train
    tr_loss += self.training_step(model, inputs)
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/src/transformers/trainer.py", line 1182, in training_step
    loss.backward()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/tensor.py", line 233, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/autograd/__init__.py", line 144, in backward
    Variable._execution_engine.run_backward(
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/zero/stage2.py", line 594, in reduce_partition_and_remove_grads
    self.reduce_ready_partitions_and_remove_grads(param, i)
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/zero/stage2.py", line 984, in reduce_ready_partitions_and_remove_grads
    self.reduce_independent_p_g_buckets_and_remove_grads(param, i)
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/zero/stage2.py", line 633, in reduce_independent_p_g_buckets_and_remove_grads
    new_grad_tensor = self.ipg_buffer[self.ipg_index].narrow(
AttributeError: 'FP16_DeepSpeedZeroOptimizer' object has no attribute 'ipg_index'

same config as in OP but changed to zero_optimization.stage = 2 +  zero_optimization.cpu_offload = true as you suggested.

I think I see the issue, based on your stack trace.
File "/mnt/nvme1/code/huggingface/transformers-deepspeed/src/transformers/trainer.py", line 1182, in training_step
    loss.backward()

Can you please call model.backward() instead of loss.backward()? I assume that model is the return value of deepspeed.initialize().

Ah, right, I switched to model.module.backward(loss) - now it wants much more gpu RAM and OOMs even bs=1. Same config as in OP. If I go back to zero_optimization.stage = 0 + zero_optimization.cpu_offload = false all is good.
Any ideas why and what do I need to change to make it work?
Talking about batch size - when does train_batch_size get used? I noticed that in all my trials until now our trainer was taking care of batching and I didn't really need to use it. However if I set it to "train_batch_size": 1 I get:
Traceback (most recent call last):
  File "./finetune_trainer.py", line 352, in <module>
    main()
  File "./finetune_trainer.py", line 273, in main
    trainer = Seq2SeqTrainer(
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/examples/seq2seq/seq2seq_trainer.py", line 57, in __init__
    super().__init__(*args, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/src/transformers/trainer.py", line 249, in __init__
    model, optimizer, training_dataloader, lr_scheduler = deepspeed.initialize(
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/__init__.py", line 110, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/engine.py", line 137, in __init__
    self._configure_with_arguments(args, mpu)
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/engine.py", line 418, in _configure_with_arguments
    self._config = DeepSpeedConfig(args.deepspeed_config,
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/config.py", line 508, in __init__
    self._configure_train_batch_size()
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/config.py", line 636, in _configure_train_batch_size
    self._batch_assertion()
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/config.py", line 575, in _batch_assertion
    assert micro_batch > 0, \
AssertionError: Micro batch size per gpu: 0 has to be greater than 0

@stas00 Thanks for the update. It appears that you running into usability gaps, so apologies for the inconvenience, but hopefully this will enable overall improvement for other users.
For zero-offload memory usage, can you please check out this #467 to see if it is relevant?
In the meantime, I will look up an explanation DeepSpeed's handling of train_batch_size et al.

Please see #131 for a discussion on batch sizes.

For zero-offload memory usage, can you please check out this #467 to see if it is relevant?

This was a perfect reference, thank you!

train_batch_size: Please see #131 for a discussion on batch sizes.

the gist of that thread was:
train_batch_size = #GPUs * train_micro_batch_size_per_gpu * gradient_accumulation_steps
So the setting I was trying was "train_batch_size": 1,:
and getting:
AssertionError: Micro batch size per gpu: 0 has to be greater than 0
I guess it divided it by 2 gpus and rounded down to 0, right?
It'd probably be a defensive tactic and saving this kind of questions - for the code to test that the derived or configured train_micro_batch_size_per_gpu > 1and assert with" train_batch_size has to be at least n_gpus*gradient_accumulation_steps"`.

and another assert that is needed that we discusses elsewhere is:

zero_optimization.cpu_offload=true, requires zero_optimization.stage=2

Going back to train_batch_size

is there a benefit to letting DS manage datasets? that is the training_data arg in deepspeed.initialize

It looks like it'd make things much more complicated for transformers to use it since we would have to re-init the ds model for each split. So it appears that letting HF trainer do the batching seems to work just fine.

I'm just not sure what to do with the `"train_batch_size":  in the config file then.
What would be your recommendations?

I will put the required asserts on my TODOs, but will appreciate a PR as well if you have the bandwidth.
There is currently no benefit to having deepspeed manage datasets, and our largest examples, bing_bert  and megatron-lm, manage their own datasets. So I think your current approach with HF trainer is fine. In future, we might explore innovations in data loading, so watch this space.
Regarding train_batch_size in the config file when deepspeed is not managing datasets, in that case it used for
(1) detecting gradient accumulation boundaries (because we expect model.step() to be called after every forward/backward), and
(2) computing aggregate throughput.
In reality, train_batch_size is not required as long as both gradient_accumulation_steps and train_micro_batch_size_per_gpu are specified.  So my suggestion is that you specify just those two, and things will automatically adjust as you scale up your GPU count, while ensuring that gradient accumulation works correctly.

I will put the required asserts on my TODOs, but will appreciate a PR as well if you have the bandwidth.

The problem is that I'm unfamiliar with those parts of code, it'd be very quick for someone who knows the code to add, and it's not urgent now that you told me what the right mix is.

There is currently no benefit to having deepspeed manage datasets, and our largest examples, bing_bert and megatron-lm, manage their own datasets. So I think your current approach with HF trainer is fine. In future, we might explore innovations in data loading, so watch this space.

A related question: what about the optimizer and lr_scheduler args of deepspeed.initialize - my guess is that they might be useful for when the user will want to supply their own if DS doesn't support what they want. But what about all the special juice that these user-supplied scheduler/optimizer will lack - i.e. do these need to be special to work with ds or any staple version would do?

Regarding train_batch_size in the config file when deepspeed is not managing datasets, in that case it used for
(1) detecting gradient accumulation boundaries (because we expect model.step() to be called after every forward/backward), and
(2) computing aggregate throughput.
In reality, train_batch_size is not required as long as both gradient_accumulation_steps and train_micro_batch_size_per_gpu are specified. So my suggestion is that you specify just those two, and things will automatically adjust as you scale up your GPU count, while ensuring that gradient accumulation works correctly.

Awesome! Thank you for the explicit details! That's very helpful!
The problem with your suggestion is this: We tweak all the cl args at the command line (and most ML programs have 100s of those), so remembering to change a config file to match the same cl args is very error-prone and will lead to a lot of mistakes/wasted time. Tweaking BS is most prevalent when dealing with OOM. So having a mismatch would be very error prone.
Could ds reuse some of the args to derive that info from  deepspeed.initialize's args - which we could adjust if need be to match the desired naming. -i.e. if I pass --batch_size=4 it'd use that as train_batch_size.
e.g. --fp16 could activate the default fp16 section, --bs batch size, etc.

A few thoughts on cl args vs. config file.

We started out passing all deepspeed args on the cl but eventually switched to config file when we realized the lack of standards in naming and semantics of cl args across client codes: e.g., train_batch_size  vs batch_size vs effective_batch_size.

Besides the difficulty of interpreting cl args, we were also concerned about adding another large number of cl args (now in the dozens and growing) to client scripts, even as we hid the associated parsing logic in deepspeed engine.

After much debate, we resorted to config file to specify parameters that meant only for the deepspeed engine, and not for the client code. There are two obvious downsides to this: (a) duplication of effort by users and (b) inconsistent values for cloned parameters. Some upsides include (a) minimal footprint/impact on client code and (b) easier to add/remove configuration parameters without breaking legacy client code. We accepted these downsides as the lesser (or easier) of two evils, especially compared to trying to interpret the meanings/intentions of cl args across client codes.

Some of our users have being bitten by this choice, especially inconsistency between cl and config file clones. But fortunately it is a one-time thing, and once they "get" it, things are smoother. But we are constantly thinking of how to make life easier. Towards that end, we recently support ability to configure deepspeed engine using a dict arg to deepspeed.initialize() instead of using a config file. With this the user can convert their cl args into the deepspeed equivalents as appropriate before calling deepspeed.initialize(). I hope this feature can be helpful to you.

Thank you for sharing your detailed notes on the design decisions with regards to the the configuration process.
I do agree that having hundreds of cl args can be difficult to manage. And your config file is a way easier to parse visually.
So far using the config file approach - I found that I can't comment things out or make comments so experimenting is somewhat difficult.

We recently support ability to configure deepspeed engine using a dict arg to deepspeed.initialize() instead of using a config file.

Oh, it's undocumented. Awesome - I'm going to try it - at the moment just passing the batch size would be fantastic. I will keep you posted once I sort it out.
p.s. I just can't say enough how much we appreciate your amazing support - it's absolutely outstanding and very helpful at quickly moving along at integrating your magic engine!

So if I try to pass train_batch_size via config_params as you suggested, it fails with:
Traceback (most recent call last):
  File "./finetune_trainer.py", line 367, in <module>
Traceback (most recent call last):
  File "./finetune_trainer.py", line 367, in <module>
    main()
  File "./finetune_trainer.py", line 282, in main
    main()
  File "./finetune_trainer.py", line 282, in main
    trainer = Seq2SeqTrainer(
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/src/transformers/trainer.py", line 258, in __init__
    trainer = Seq2SeqTrainer(
  File "/mnt/nvme1/code/huggingface/transformers-deepspeed/src/transformers/trainer.py", line 258, in __init__
    model, optimizer, training_dataloader, lr_scheduler = deepspeed.initialize(
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/__init__.py", line 110, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/engine.py", line 138, in __init__
    model, optimizer, training_dataloader, lr_scheduler = deepspeed.initialize(
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/__init__.py", line 110, in initialize
    self._do_sanity_check()
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/engine.py", line 449, in _do_sanity_check
    engine = DeepSpeedEngine(args=args,
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/engine.py", line 138, in __init__
    assert self._is_supported_optimizer(self.optimizer_name()), \
      File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/engine.py", line 444, in _is_supported_optimizer
self._do_sanity_check()
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/engine.py", line 449, in _do_sanity_check
    getattr(torch.optim, optimizer_name, None) is not None
TypeError: getattr(): attribute name must be string
    assert self._is_supported_optimizer(self.optimizer_name()), \
  File "/mnt/nvme1/code/github/00optimize/DeepSpeed/deepspeed/runtime/engine.py", line 444, in _is_supported_optimizer
    getattr(torch.optim, optimizer_name, None) is not None
TypeError: getattr(): attribute name must be string

And another question:
We have:
args.per_device_eval_batch_size
args.per_device_train_batch_size

which can be quite different. How should we handle this with deepspeed? As I see there is only one batch_size var and its train_batch_size - no variation for eval.
The evaluation is in general less memory demanding due to not needing gradients, yet it can be quite memory hungry due to beam search and require more memory than training.

Yes, DeepSpeed currently does not support eval/inference execution. The reason is that training was previously the memory and computation intensive part. And so our current examples do evaluation outside of DeepSpeed. But recently, we have seen requests to add support for evaluation/inference.  So it is now on our short-term timeline.

Oh, I was completely unaware of this. So I need to do more work then to undo deepspeed post-training stage.
I suppose that I need to remove DeepSpeed layer from DDP(DeepSpeed(OriginalModel)) stack, so it becomes DDP(OriginalModel) as soon as training is done. So something like:
# remove Deepspeed from the stack
self.model_wrapped.module = self.mode_wrapped.module.module
gc.collect() # force memory reclaim immediately

Also need to make sure that there are no memory-holding left-overs from deepspeed, so that all of GPU is available again.
You don't think parts of ZeRO could be of help during eval? I guess there aren't many moving parts. I'm thinking perhaps memory management could save some memory there. But I could be wrong.

@stas00 I typically just retain reference to my original nn.Module in memory while operating with my model_engine (also an nn.Module), and simply perform validation on my validation set with the reference to the original module.
DeepSpeedEngine only wraps around the original nn.Module, so validation works just fine with the reference to the original module.

Thank you for your comments, @g-karthik.
Yes, we are recoding things now so that we have:

self.model - original
self.model_wrapped - wrapped
so that it's obvious which is which and easily available on demand.

I suspect we must clear out any references to deepspeed to free the memory if we aren't using it at later stages, but I could be wrong and it already happens automatically. I will discover that once I get a chance to implement it as I didn't realize I needed to remove DS for the evaluation stage, since it worked just fine with it.

@stas00 I think you'll probably need to free some references for evaluation when the model size is very large, such as perhaps T5-11B. We can discuss this on your transformers PR, feel free to tag me there so I get an email.

@g-karthik Thanks for sharing your experience with using DeepSpeed for evaluation. @stas00, I will gladly defer to @g-karthik on this topic, as he is quite more knowledgeable than me :).

We are much appreciating you too offering to support our DS integration process, @g-karthik!

@tjruwase, I got a chance to experiment with your suggestions and it is mostly working with some TLC needed for deepspeed.initialize.

I cannot not pass deepspeed_config in args - it crashes. Since you said I should be able to send all the config via config_params, why does it still require deepspeed_config? And the precedence is unclear then should config_params provide different values

it requires I pass local_rank via args too. I guess it's fine. I thought passing all args via config_params is what was expected but it's no problem. It's just not super clear what goes in args and what in config_params. And the first one expects an obj (due to argparse), the second a dict. Not a problem either, a learning curve...

I no longer ask to be able to pass the config file and the overrides, since I have to read the config file in anyway to check that the user hasn't preset anything that we are going to override to avoid errors. So I'm totally fine with just passing a config dict that I sort out myself.

At the moment here is what I came up with.
(Reminding I'm trying to solve the problem of needing to get some config values through HF trainer cl args, but the bulk of it via ds_config.js)
    def _init_deepspeed(self, model):
        import io, json
        from types import SimpleNamespace

        # for clarity extract what args are being passed to deepspeed
        ds_args = {k: getattr(self.args, k, None) for k in ["deepspeed_config", "local_rank"]}

        with io.open(self.args.deepspeed_config, 'r', encoding='utf-8') as f:
            config = json.load(f)

        # The following code injects some of trainer's cl args into the DS config

        # First to ensure that there is no mismatch between cl args values and presets in the config
        # file, asking to not set "train_batch_size", "train_micro_batch_size_per_gpu",
        # "gradient_accumulation_steps" in ds config file
        bs_keys = ["train_batch_size", "train_micro_batch_size_per_gpu"]
        if len([x for x in bs_keys if x in config.keys()]):
            raise ValueError(f"Do not include {bs_keys} entries in the ds config file, as they will be set via --per_device_train_batch_size or its default")
        if "gradient_accumulation_steps" in config.keys():
            raise ValueError(f"Do not include gradient_accumulation_steps entries in the ds config file, as they will be set via --gradient_accumulation_steps or its default")

        # DeepSpeed does:
        #   train_batch_size = n_gpus * train_micro_batch_size_per_gpu * gradient_accumulation_steps
        # therefore we just need to set:
        #   train_micro_batch_size_per_gpu == args.per_device_train_batch_size
        #   gradient_accumulation_steps    == args.gradient_accumulation_steps
        # and it will figure out the rest
        config["train_micro_batch_size_per_gpu"] = self.args.per_device_train_batch_size
        config["gradient_accumulation_steps"] = self.args.gradient_accumulation_steps

        # init that takes some config via `args`, and the bulk of it via `config_params`
        model_parameters = filter(lambda p: p.requires_grad, model.parameters())
        model, optimizer, _, lr_scheduler = deepspeed.initialize(
            args=SimpleNamespace(**ds_args),  # expects an obj
            model=model,
            model_parameters=model_parameters,
            config_params=config,
        )

        return model, optimizer, lr_scheduler

@stas00 Happy New Year. Apologies for the radio silence, I finally succumbed to the holidays.
I want to resume the integration effort starting with this issue of deepspeed.initialize required args.deepspeed_config. This shouldn't be the case, and so I am investigating now.

@stas00 Happy New Year. Apologies for the radio silence, I finally succumbed to the holidays.
I want to resume the integration effort starting with this issue of deepspeed.initialize required args.deepspeed_config. This shouldn't be the case, and so I am investigating now.

Happy New Year to you too, @tjruwase! Yes, it is hard to do co-operative work when those one cooperates with aren't there ;)

Happy New Year to you too, @tjruwase! Yes, it is hard to do co-operative work when those one cooperates with aren't there ;)

I just submitted a fix and unit test. Thanks for catching this bug.

I just submitted a fix and unit test. Thanks for catching this bug.

I checked that it is no longer required - thank you for the quick fix.

I checked that it is no longer required - thank you for the quick fix.


RTX腾讯通 企业内部的聊天软件

NOTE: Each discrete (stand-alone) request should be in its own issue.
Put "Add ___" as Title
RTX_腾讯通企业内部聊天软件
Provide a link to the proposed addition.
https://www.rtxapp.com/mac/
Explain why it should be added
还是有一部分企业内部在使用此通讯工具的，这个软件有Mac版本也是win版本，连接是Mac版本的。
Additional context

Issue Checklist
Put an x in the boxes once you've completed each step. You can also fill these out after creating the issue. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before responding to the issue.

[ ] I have checked for other similar issues
[ ] I have explained why this change is important
[ ] I have added necessary documentation (if appropriate)

@aliliin 欢迎提一个 PR，我会将它合并。谢谢！

@jaywcjlove 已经提交。


How to connect bloomfilter via php

Environment info:

KrakenD version: 2.0.6
System info: Linux <IP_ADDRESS>-microsoft-standard-WSL2 x86_64
Backend technology: PHP

Describe what are you trying to do:
I want to revoke jwt token via my backend(lumen) written in php. How do I connect to krakend bloomfilter and revoke the token?
Your configuration file:

{
  "$schema": "https://www.krakend.io/schema/v3.json",
  "version": 3,
  "name": "ocean-chat",
  "timeout": "60000ms",
  "cache_ttl": "300s",
  "output_encoding": "json",
  "read_timeout": "60s",
  "write_timeout": "60s",
  "idle_timeout": "60s",
  "read_header_timeout": "60s",
  "plugin": {
    "pattern": ".so",
    "folder": "/opt/krakend/plugins/"
  },
  "extra_config": {
    "router": {
      "return_error_msg": true
    },
    "auth/revoker": {
      "hash_name": "default",
      "N": 10000000,
      "P": 1e-7,
      "port": 1234,
      "token_keys": [
        "jti"
      ],
      "TTL": 2628288,
      "revoke_server_ping_url": "http://localhost:8080/instances",
      "revoke_server_ping_interval": "30s",
      "revoke_server_api_key": "639ee23f-f4c5-40c4-855c-912bf01fae86",
      "revoke_server_max_workers": 5
    }
  },
  "endpoints": [
    {
      "endpoint": "/auth/register",
      "method": "POST",
      "backend": [
        {
          "url_pattern": "/api/register",
          "extra_config": {
            "backend/http": {
              "return_error_code": true
            }
          },
          "sd": "static",
          "method": "POST",
          "host": [
            "http://nginx:81"
          ]
        }
      ]
    },
    {
      "endpoint": "/auth/verify-otp",
      "method": "POST",
      "backend": [
        {
          "url_pattern": "/api/verify-otp",
          "extra_config": {
            "backend/http": {
              "return_error_code": true
            }
          },
          "sd": "static",
          "method": "POST",
          "host": [
            "http://nginx:81"
          ]
        }
      ]
    },
    {
      "endpoint": "/auth/login",
      "method": "POST",
      "backend": [
        {
          "host": [
            "http://nginx:81"
          ],
          "url_pattern": "/api/login",
          "sd": "static",
          "method": "POST",
          "extra_config": {
            "backend/http": {
              "return_error_code": true
            }
          }
        }
      ],
      "extra_config": {
        "auth/signer": {
          "alg": "HS256",
          "kid": "user",
          "keys_to_sign": [
            "access_token",
            "refresh_token"
          ],
          "jwk_url": "http://nginx:81/api/jwk",
          "disable_jwk_security": true
        }
      }
    },
    {
      "endpoint": "/auth/logout",
      "method": "POST",
      "backend": [
        {
          "host": [
            "http://nginx:81"
          ],
          "url_pattern": "/api/logout",
          "sd": "static",
          "method": "POST",
          "extra_config": {
            "backend/http": {
              "return_error_code": true
            }
          }
        }
      ],
      "extra_config": {
        "auth/validator": {
          "alg": "HS256",
          "jwk_url": "http://nginx:81/api/jwk",
          "disable_jwk_security": true,
          "operation_debug": true,
          "cache": false
        }
      }
    }
  ]
}

Configuration check output:
arsing configuration file: krakend.json
^C2023/03/18 08:41:57 Signal intercepted: interrupt
Global settings
        Name: ocean-chat
        Port: 8080
        Folder: /opt/krakend/plugins/
        Pattern: .so
2 global component configuration(s):
- auth/revoker
- router
4 API endpoint(s):
        - POST /auth/register
        Timeout: 1m0s
        Connecting to 1 backend(s):
                [+] POST /api/register
                Timeout: 1m0s
                Hosts: [http://nginx:81]
                1 backend component configuration(s):
                - backend/http

        - POST /auth/verify-otp
        Timeout: 1m0s
        Connecting to 1 backend(s):
                [+] POST /api/verify-otp
                Timeout: 1m0s
                Hosts: [http://nginx:81]
                1 backend component configuration(s):
                - backend/http

        - POST /auth/login
        Timeout: 1m0s
        1 endpoint component configuration(s):
        - auth/signer
        Connecting to 1 backend(s):
                [+] POST /api/login
                Timeout: 1m0s
                Hosts: [http://nginx:81]
                1 backend component configuration(s):
                - backend/http

        - POST /auth/logout
        Timeout: 1m0s
        1 endpoint component configuration(s):
        - auth/validator
        Connecting to 1 backend(s):
                [+] POST /api/logout
                Timeout: 1m0s
                Hosts: [http://nginx:81]
                1 backend component configuration(s):
                - backend/http

0 async agent(s):
Syntax OK!

Commands used:
"run", "-d", "-c", "/etc/krakend/krakend.json" (docker)

from the documentation (https://www.krakend.io/docs/authorization/revoking-tokens/#bloom-filter-client):

The communication with the bloom filter is RPC-based. The component exposes a listening port of your choice to receive updates of the bloom filter (single or batch), but a client is needed to communicate with the component.
When using the open-source edition, you have to build your client. Look at the bloom filter library, which includes a client. In addition, the KrakenD Playground project consists of a sample web page with a form and an RPC client that sends commands to the bloom filter and updates it.
When using the Enterprise edition the Revoke Server connects to all KrakenD instances as a client, and there’s nothing you need to build to make it work.

Cheers!


Add this helm repo to chartcenter.io

@odavid would you mind to open a PR in https://github.com/jfrog/chartcenter/blob/master/repos.yaml, so your helm repo
is officially part of the chartcenter.io?
chartcenter.io is the new central helm charts repo which even can be used from the helm client :)

Hi @rimusz - See https://github.com/jfrog/chartcenter/pull/17
Cheers


Constructor Uint8Array requires 'new' when targeting ES5

TypeScript Version<EMAIL_ADDRESS>Search Terms:
Constructor Uint8Array requires 'new'
TypedArray requires 'new'
Uint8Array requires 'new'
Code
{
	"compilerOptions": {
		"target": "es5",
		"lib": [ "es5", ],
	}
}

abstract class Bytes extends Uint8Array {
	static factory<T extends typeof Bytes & { new() }>(this: T): InstanceType<T> {
		return (new this()) as InstanceType<T>
	}
}

class Bytes4 extends Bytes {
	constructor() { super(4) }
}

const bytes4 = Bytes4.factory()

Expected behavior:
Either a compiler error giving me better insight into why I cannot instantiate a Uint8Array in this way, or it working at runtime.
Actual behavior:
At runtime:
TypeError: Constructor Uint8Array requires 'new'
source-map-support.js:444
    at Bytes4.Uint8Array (<anonymous>)
    at Bytes4.Bytes [as constructor] (.\test\test.ts:17:42)
    at new Bytes4 (.\test\test.ts:8:18)
    at Function.Bytes.factory (.\test\test.ts:3:11)
    at Object.<anonymous> (.\test\test.ts:11:23)
    at Module._compile (internal/modules/cjs/loader.js:686:14)
    at Module.m._compile (.\test\node_modules\ts-node\src\index.ts:413:23)
    at Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Object.require.extensions.(anonymous function) [as .ts] (.\test\node_modules\ts-node\src\index.ts:416:12)
    at Module.load (internal/modules/cjs/loader.js:599:32)

Playground Link: Playground
Related Issues:
https://github.com/Microsoft/TypeScript/issues/24195
https://github.com/Microsoft/TypeScript/issues/1168
Note: I don't actually understand the error, so I apologize if this isn't actually a bug in TypeScript.  I think it has to do with Uint8Array being special and not actually a class (even though it looks like a class).  You can do new Uint8Array(4) and new Uint8Array() in node and browser, but for some reason you cannot do Uint8Array.apply({}, 4).
Note2: This works if you target ES2015 or higher.

You cannot extend built-ins in ES5.  Classes in theory don't exist in ES5 (let alone Uint8Arrays) and so TypeScript tries to down emit items that look like classes in a way that is not compatible with extending built in.  When you target ES2015 and beyond, TypeScript doesn't do a down emit, and therefor the extension is passed directly to the runtime and does not cause an error.
There is no logic in TypeScript to be able to differentiate between a built in and any other class, therefore TypeScript always tries to down emit the extension the same way.  If the Uint8Array was a polyfill, because you were really running in a ES5 environment, the extension would likely work.

Oh, so the real problem here is that Uint8Array should not be available in lib: [ "es5" ] and I should have received a compiler error telling me as much?

No it exists. The issue is that there's no way to correctly extend built-ins without Reflect.construct which is ES6.

According to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array it wasn't added until es2015.

While it wasn't added to the language specification until ES2015, it was present in IE10+ and most other browsers, so TypeScript has traditionally included it in the ES5 typings.  It is highly unlikely that you would find an ES5 environment that didn't support it.  Still, all built ins were not extensible in ES5.  It was something that was only addressed in ES2015.

Is there some reason that the compiler can't error/warn when someone tries to extend a built-in while targeting ES5?

TypeScript has no concept of what is a built in and what isn't.
It would be a guess on my part, implementing a system would be complicated and only benefit a very narrow use case (extending builtins when targeting ES3/ES5).


Recognize single hour lead times.

The changes to the github actions look good to me.
I am not familiar with the project so I can not comment much on the changes of the atab.
Is there a test that covers the functionality of the atab changed ? Is it integrated with jenkins, so could you run the test to check it for this PR ?


update control panel layout

Regarding https://github.com/plone/volto/issues/4789

@ksuess I need to keep working on it, for the tests to pass, and to implement the checkbox change. I will work on it on Monday, but feel free to make any changes in the meantime!

@danalvrz Looks good to me, but needs some work to be responsive. Can you fix this? Thanks.

@danalvrz Please rebase your branch, I've made a change in mine https://github.com/plone/volto/tree/relations-control-panel

@ksuess yes, I'll fix the responsiveness and rebase

@ksuess I removed the sticky property for the labels because it was breaking the table's layout when horizontally scrolling. But I can revert that change.
Screenshot

That looks much better.
I suggest to work with the grid gap to achieve the distance between source options and target options.

Well, it is a challenge, but the sticky labels of the targets is in my eyes important to be able to manage the relations.

For testing purpose, to create a lot of content with a lot of relations, you can create example conten with:

That looks much better. I suggest to work with the grid gap to achieve the distance between source options and target options.

Using the grid gap was the right call!


config_param options to specify deprecated/obsoleted parameters

This change adds 2 options for config_param.
And I marked host, port of out_forward without <server> sections as obsoleted.
deprecated
Unrecommended parameter, and will be removed in future.
If users specify this parameter, Fluentd warns it in logs.
obsoleted
Configuration parameter which is already removed.
If users specify this parameter, Fluentd stops with an error (a kind of configuration errors).

@repeatedly could you review this change?

One idea. How about adding "message" like parameter to show better logs?.
 config_param :host, :string, default: nil, obsoleted: true, message: "Use <server> section instead"

result:
 'host' parameter is already removed. Don't use it. Use <server> section instead"

@repeatedly When is that parameter used? only for deprecated and obsoleted?
If so, it may be better to change value type of these options from true/false to string of message to show. Isn't it?

it may be better to change value type of these options from true/false to string of message to show. Isn't it?

LGTM 👍

Good. After test passed, please merge.

Merged.


update golang test images for calico and cilium

What this PR does / why we need it:
Update golang test images for calico and cilium.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:

/kind enhancement
/lgtm
/approve


Update manage-resource-accounts.md

https://github.com/MicrosoftDocs/OfficeDocs-SkypeForBusiness/issues/2539
Added note.

@nenonix @JohanFreelancer9 Copy editing is needed for this PR. Thanks.

This information is no longer relevant, the licensing requirement has changed.


fix #12 ActionButton positioning

Any particular reason to set the height & width of the container explicitly ? Just setting absolute values (e.g: top: 0) should be enough.
I think Dimensions.get('window').height gives you the height of the whole window & not just the view available to the app, which is why when the state is active, the height of the container gets a bit larger and overflows.
This should also fix the issue on landscape mode here: https://github.com/mastermoo/react-native-action-button/issues/12

Thanks @arshsingh
Did you check this on Android. Not setting height and width is an issue and the "bottom" property doesn;t work ..

This change works almost perfect for me on Android except if the button is centered and in landscape, then it's still off. However, it seems like this might be an issue with React Native itself. See: https://github.com/facebook/react-native/issues/3264

Thanks. Will check it out and merge.
Cheers
On Dec 27, 2015 11:42 AM, "nvladimiroff"<EMAIL_ADDRESS>wrote:

This change works almost perfect for me on Android except if the button is
centered and in landscape, then it's still off. However, it seems like this
might be an issue with React Native itself. See:
facebook/react-native#3264
https://github.com/facebook/react-native/issues/3264
—
Reply to this email directly or view it on GitHub
https://github.com/mastermoo/react-native-action-button/pull/14#issuecomment-167388431
.

@arshsingh If I remove { height : sH, width : sW }  this is what I get on Android.


IP whitelist does not work for a specific IP

Bug description
IP whitelist does not work for a specific IP
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[X] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
IP whitelist works for a specific IP
Steps to reproduce the bug
Whitelist works normally for the following configs:
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: whitelistip
spec:
  compiledAdapter: listchecker
  params:
    overrides: ['<IP_ADDRESS>/16']
    blacklist: false
    entryType: IP_ADDRESSES

---
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: sourceip
spec:
  compiledTemplate: listentry
  params:
    value: request.headers["remoteip"]

---
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: checkip
spec:
  match: source.labels["istio"] == "ingressgateway"
  actions:
    - handler: whitelistip
      instances: [sourceip]

But not when I change the ip range to a specific one:
...
spec:
  compiledAdapter: listchecker
  params:
    overrides: ['123.321.123.321']
    blacklist: false
    entryType: IP_ADDRESSES
...

Version (include the output of istioctl version --remote and kubectl version)
~ istioctl version --remote
client version: 1.2.5
grafana version:
grafana version:
citadel version: 1.2.5
galley version: 1.2.5
ingressgateway version: 1.2.5
pilot version: 1.2.5
pilot-gateway version:
policy version: 1.2.5
policy-gateway version:
sidecar-injector version: 1.2.5
telemetry version: 1.2.5
telemetry-gateway version:
kiali version:
prometheus version:

~ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T19:44:19Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-aliyun.1", GitCommit:"8cb561c", GitTreeState:"", BuildDate:"2019-05-23T11:28:17Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
Aliyun ACK, Operator
Environment where bug was observed (cloud vendor, OS, etc)
Aliyun ACK, CentOS 7

Neither works for ip range like:
    overrides: ['123.321.123.321/24']

123.321.123.321 is not a valid ip address neither is 123.321.123.321/24 321 >255

@ddebasilio Cos I don't want to expose the real public ip here.

Maybe you can check the codes here, and write a test with you own public ip.
https://github.com/istio/istio/blob/e4f1f0fdbb2e8e7c70078931e235bd26c5e93ad8/mixer/adapter/list/ipList.go#L36-L60

@waret Thanks for ur reply, but sorry I'm not familiar with go.
Could any guy help to tell what is wrong with my ip format?

@xiaozhongliu Can you "ping" from the whitelisted IP address to the host server? My guess is that this is not related to the ip but to the firewall setup itself. I don't see anything wrong with the code waret linked to.

@xiaozhongliu : Couple of things to help you get started debugging this?

Can you enable access logs(https://istio.io/docs/tasks/telemetry/logs/collecting-logs/) and see traffic is actually flowing through Istio?
Also, can you add attribute request.headers["remoteip"] in access log and see its value is same as the ip you whitelisted as?
You can also looks at the logs of policy pod in your istio install to see any red herrings with your setup.

@gargnupur Thanks for ur suggestions, I can clearly see the rejected ip address:

This is the normal situation I described in the original post:
Whitelist works normally for this ip range: ['<IP_ADDRESS>/16']
But not for specific public ip or public ip range:
<IP_ADDRESS>
['<IP_ADDRESS>/24']

I tried public ip in https://github.com/istio/istio/blob/master/tests/integration/mixer/policy/white_black_listing_test.go integration test and it works...
Anything else specific to your setup?

@gargnupur Thanks for replying, what config yaml files did you use, could you share with me?

I used this one and changed the ip to the one you suggested: https://github.com/istio/istio/blob/master/samples/bookinfo/policy/mixer-rule-deny-ip.yaml


python3Packages.seasonal: init at 0.3.1

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD" on x86_64-linux:
2 packages built:
python310Packages.seasonal python311Packages.seasonal

@GrahamcOfBorg build python310Packages.seasonal python311Packages.seasonal

Dependency of https://github.com/NixOS/nixpkgs/issues/247089

@natsukium all fixed!

Result of nixpkgs-review pr 256348 run on x86_64-linux 1

4 packages built:

python310Packages.seasonal
python310Packages.seasonal.dist
python311Packages.seasonal
python311Packages.seasonal.dist

@GrahamcOfBorg build python310Packages.seasonal python311Packages.seasonal


fixed ObjectAggregationIterator#createNewAggregationBuffer  repeat computer bufferFieldTypes

What changes were proposed in this pull request?

ObjectAggregationIterator's createNewAggregationBuffer function repeat to computer bufferFieldTypes for same inputs, this invoke high occupation ratio on  Flame Graph

Why are the changes needed?

Does this PR introduce any user-facing change?

provides an aggregateBufferFieldTypes property that does not need to be calculated each time.
How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Mind filien a JIRA please?

NO
At 2024-11-12 12:56:14, "Hyukjin Kwon" @.***> wrote:
Mind filien a JIRA please?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Mind filien a JIRA please?

Request a Jira account,i have request a Jira account with email.