# @sshleifer Progress Update Aug 4 -\> Aug 19 **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 19, 2020, 2:14pm UTC](https://discuss.huggingface.co/t/sshleifer-progress-update-aug-4-aug-19/767 "2020-08-19T14:14:50Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 19, 2020, 2:14pm UTC](https://discuss.huggingface.co/t/sshleifer-progress-update-aug-4-aug-19/767/1 "2020-08-19T14:14:50Z")
In the spirit of openness (and inspired by sylvain’s google doc) I’m posting my standup/progress update openly instead of on google docs. Big thank yous to @sgugger for 35 code reviews, @julien-c for moonlanding help support with marian, and @valhalla and @stas for help in so many areas over the past few weeks. Successes/Done: - [#6340](https://github.com/huggingface/transformers/pull/6340) 12 new pegasus checkpoints - SOTA summarization on many new datasets. - [#6342](https://github.com/huggingface/transformers/pull/6342) 357 new MARIAN checkpoints, 48 **upgrades** = our 1st experiment with model versioning. conversion code uses very rough heuristics to **check** whether we have an identical model. (across naming schemes), and either converts, upgrades, or ignores based on BLEU score posted in README.MD. This also added very involved, automated model-carding. - [#6526](https://github.com/huggingface/transformers/pull/6526) Partially undo adjust\_logits\_during\_generation hack: metrics improvements across the board. We now are pretty close to metric parity. - Successfully distilled mbart english to romanian. It works on MT! see `from_pretrained(sshleifer/distilbart-enro-12-6)` or 12-4. - Lots of DM onboarding/QA with stas. He is porting fairseq’s SOTA translator now, after many smaller helpful bug fixes. - Lots of maintenance while others were on vacation. - Secret support mission going well. Struggles: - I am close (4 common tests failing, integration test passing) on [tfbart](https://github.com/huggingface/transformers/pull/5411), could use some help. I think I need to do the huge signature like `TFT5ForConditionalGeneration`. After this, tfpegasus, tfmbart etc. will be easier. - Our Seq2Seq dataloader is slower than fairseq’s because it doesn’t use dynamic batch size. - external contributors cannot get TPU+seq2seq working, I may take a pass. - After huge amount of effort, many 24h experiments, our Seq2Seq finetune metrics are still **slightly** below fairseq, (0.2 ROUGE xsum, 1 BLEU wmt\_en\_ro). I suspect this is the dataloader. - Slow GPU CI has been broken for 10 days. - Haven’t carved out time for research in a while. Future: 1. get marian, pegasus tweeted and “released”. Support as needed. 2. [tfbart](https://github.com/huggingface/transformers/pull/5411) 3. CARVE OUT RESEARCH TIME: freezing idea. 4. seq2seq dataloader - dynamic batch size, cleanup to always use `prepare_seq2seq_batch` 5. seq2seq finetune onTPU (through `Trainer` !) ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [August 19, 2020, 2:26pm UTC](https://discuss.huggingface.co/t/sshleifer-progress-update-aug-4-aug-19/767/2 "2020-08-19T14:26:45Z")
Already sounds like great progress! I’m very interested in the seq2seq stuff these days, and I miss customizability in onmt a bit - I did some efforts, but everything is very interconnected (data and models)) so customization is not very straightforward - so having great seq2seq in transformers would be helpful for me. About SOTA on summarization; are you planning to jot down your findings in a paper or blogpost? Might be an interesting read! ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 19, 2020, 2:51pm UTC](https://discuss.huggingface.co/t/sshleifer-progress-update-aug-4-aug-19/767/3 "2020-08-19T14:51:32Z")
Not really my findings, just the pegasus paper and a bug fix. We used to be forcing Bart and all it’s children (Mbart, Pegasus) to generate bos after `decoder_start_token_id`, because fairseq does that. With @valhalla 's help, we figured out that this only helps for `bart-large-cnn`, so I added a config attribute `force_bos_token_to_be_generated`, which defaults to False, but is set to true for bart-large-cnn and `distilbart-cnn*`. #### Metrics Impact `facebook/bart-large-xsum` : + 0.6 ROUGE2, 22.38 `facebook/mbart-large-enro`: +0.3 BLEU, 28.15 `google/pegasus-xsum`: +.3 ROUGE2, 24.43 `facebook/bart-large-cnn`: 0 ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 19, 2020, 3:04pm UTC](https://discuss.huggingface.co/t/sshleifer-progress-update-aug-4-aug-19/767/4 "2020-08-19T15:04:15Z")
> get marian, pegasus tweeted and “released”. Support as needed. is latest release coming soon ? Let me when to tweet . ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 19, 2020, 3:07pm UTC](https://discuss.huggingface.co/t/sshleifer-progress-update-aug-4-aug-19/767/5 "2020-08-19T15:07:50Z")
@sshleifer Thanks to you for answering all of my questions timely ! ---
**Author:** ![stas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stas/32/19289_2.png) [@stas](https://discuss.huggingface.co/u/stas)\ **Post date:** [August 19, 2020, 4:00pm UTC](https://discuss.huggingface.co/t/sshleifer-progress-update-aug-4-aug-19/767/6 "2020-08-19T16:00:21Z")
Me too - I’ve been getting an amazing and caring support from @sshleifer! So appreciating you, Sam!