# Remove a named module from a pre-trained model

**URL:** <https://discuss.huggingface.co/t/remove-a-named-module-from-a-pre-trained-model/81477>\
**Category:** 🤗Transformers\
**Created:** [April 12, 2024, 6:19pm UTC](https://discuss.huggingface.co/t/remove-a-named-module-from-a-pre-trained-model/81477 "2024-04-12T18:19:52Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![celalp](https://avatars.discourse-cdn.com/v4/letter/c/8edcca/32.png) [@celalp](https://discuss.huggingface.co/u/celalp)\
**Post date:** [April 12, 2024, 6:19pm UTC](https://discuss.huggingface.co/t/remove-a-named-module-from-a-pre-trained-model/81477/1 "2024-04-12T18:19:52Z")

</div>

Hi,

I’m trying to remove a pooling layer that is at the bottom of my pre trained model.

```auto
BertModel(
  (embeddings): BertEmbeddings(
    (word_embeddings): Embedding(30, 1024, padding_idx=0)
    (position_embeddings): Embedding(40000, 1024)
    (token_type_embeddings): Embedding(2, 1024)
    (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.0, inplace=False)
  )
  (encoder): BertEncoder(
    (layer): ModuleList(
      (0-29): 30 x BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=1024, out_features=1024, bias=True)
            (key): Linear(in_features=1024, out_features=1024, bias=True)
            (value): Linear(in_features=1024, out_features=1024, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=1024, out_features=1024, bias=True)
            (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.0, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=1024, out_features=4096, bias=True)
          (intermediate_act_fn): GELUActivation()
        )
        (output): BertOutput(
          (dense): Linear(in_features=4096, out_features=1024, bias=True)
          (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.0, inplace=False)
        )
      )
    )
  )
  (pooler): BertPooler(
    (dense): Linear(in_features=1024, out_features=1024, bias=True)
    (activation): Tanh()
  )
)

```

I want to remove/replace the final BertPooler layer but keep everything the same (including the pre-trained weights). Is there a way to accomplish this?

Thanks