# Deploying Huggingface Sagemaker Models with Elastic Inference **URL:** \ **Category:** Amazon SageMaker\ **Created:** [July 27, 2021, 5:04pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736 "2021-07-27T17:04:16Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![schopra](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@schopra](https://discuss.huggingface.co/u/schopra)\ **Post date:** [July 27, 2021, 5:04pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/1 "2021-07-27T17:04:16Z")
When I try to deploy a HuggingFace Sagemaker model with elastic inference (denoted by the `accelerator_type` parameter) I get an error. **Deploy Snippet:** ```auto predictor = huggingface_model.deploy( initial_instance_count=1, instance_type="ml.t2.medium", accelerator_type='ml.eia2.medium' ) ``` **Error Msg:** ```auto ~/miniconda3/envs/ner/lib/python3.8/site-packages/sagemaker/image_uris.py in _validate_arg(arg, available_options, arg_name) 305 """Checks if the arg is in the available options, and raises a ``ValueError`` if not.""" 306 if arg not in available_options: --> 307 raise ValueError( 308 "Unsupported {arg_name}: {arg}. You may need to upgrade your SDK version " 309 "(pip install -U sagemaker) for newer {arg_name}s. Supported {arg_name}(s): " ValueError: Unsupported image scope: eia. You may need to upgrade your SDK version (pip install -U sagemaker) for newer image scopes. Supported image scope(s): training, inference. ``` The model deploys successfully if I do not provide an accelerator (i.e., no Elastic Inference). Do the HuggingFace Sagemaker models support EI? If yes, how might I deploy the model successfully with EI? And if not, is EI support on the roadmap? Much thanks in advance! 😄 ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [July 28, 2021, 6:46am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/2 "2021-07-28T06:46:03Z")
Hey @schopra, Sadly speaking we don’t have `EI` DLCs yet. We are working on it and it is on the roadmap with one of the highest priorities. I would update this thread here when I got any news. ---
**Author:** ![ujjirox](https://avatars.discourse-cdn.com/v4/letter/u/ccd318/32.png) [@ujjirox](https://discuss.huggingface.co/u/ujjirox)\ **Post date:** [September 3, 2021, 4:35pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/3 "2021-09-03T16:35:06Z")
Is there by any chance a list of supported instances at this time? Thanks! ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [September 5, 2021, 9:32am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/4 "2021-09-05T09:32:03Z")
Hey @ujjirox, supported instances for what? Training or Inference or both? You can find an overview of supported instances type for sagemaker here: [Amazon SageMaker Pricing – Amazon Web Services (AWS)](https://aws.amazon.com/sagemaker/pricing/?nc1=h_ls) ---
**Author:** ![ujjirox](https://avatars.discourse-cdn.com/v4/letter/u/ccd318/32.png) [@ujjirox](https://discuss.huggingface.co/u/ujjirox)\ **Post date:** [September 5, 2021, 4:14pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/5 "2021-09-05T16:14:39Z")
Sorry! Should have been more clear. I meant for inference. I actually had tried running inference with ml.inf1.xlarge but it didn’t seem to work, hence the question. Thanks. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [September 6, 2021, 6:19am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/6 "2021-09-06T06:19:01Z")
Hey @ujjirox, Inferentia is also not yet supported, since we need to create a separate DLC for the Inferentia instances, but we are on it. Other than this every CPU / GPU machine should be supported. ---
**Author:** ![asafab](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@asafab](https://discuss.huggingface.co/u/asafab)\ **Post date:** [December 28, 2021, 1:22pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/7 "2021-12-28T13:22:30Z")
Hey, any news regarding the EI DLC/ INF DLC? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [December 29, 2021, 10:21am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/8 "2021-12-29T10:21:21Z")
Hey @asafab, Yes, i already opened a PR for INF DLCs: You can follow it here: [[huggingface\_pytorch][NEURON][build] Huggingface Neuron inference DLC by philschmid · Pull Request #1578 · aws/deep-learning-containers · GitHub](https://github.com/aws/deep-learning-containers/pull/1578) When it is merged and available we will additionally share on social media + provide an example. ---
**Author:** ![asafab](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@asafab](https://discuss.huggingface.co/u/asafab)\ **Post date:** [December 29, 2021, 10:35am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/9 "2021-12-29T10:35:44Z")
very cool! thanks for the response, you have news about the EI DLC too? ---
**Author:** ![YannAgora](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@YannAgora](https://discuss.huggingface.co/u/YannAgora)\ **Post date:** [January 27, 2022, 10:38am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/10 "2022-01-27T10:38:16Z")
Hey there !! The PR regarding INF DLCs seems to have been merged, does it mean `ml.inf*` instance family can now be used with HuggingFace models ? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [January 27, 2022, 10:56am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/11 "2022-01-27T10:56:53Z")
They are merged but yet not released. I hope they will be available in the next 2 weeks. We ll let you know on social media. ---
**Author:** ![YannAgora](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@YannAgora](https://discuss.huggingface.co/u/YannAgora)\ **Post date:** [March 18, 2022, 3:50pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/12 "2022-03-18T15:50:42Z")
Hello @philschmid, I read your article [https://www.philschmid.de/huggingface-bert-aws-inferentia](https://www.philschmid.de/huggingface-bert-aws-inferentia) about hugging face model deployment on inferentia instance (very good and clear btw). Can this method be used for all model types and all tasks ? I particularly think to Seq2Seq models (Bart, Pegasus, T5) for Summarization task ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [March 21, 2022, 12:53pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/13 "2022-03-21T12:53:17Z")
Hello @YannAgora, Yes, it can also be used for T5 or pegasus. You can find more documentation here: [Transformers MarianMT Tutorial — AWS Neuron documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/transformers-marianmt.html#GenerationMixin-Class). You can use the `NeuronGeneration` code inside the `inference.py` then. ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [May 18, 2022, 11:42am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/14 "2022-05-18T11:42:52Z")
Hi @philschmid Any updates on EI (Elastic Inference) DLCs for inference? Can we start using EI accelerators with HuggingFace models for inference? Thanks ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [May 18, 2022, 1:10pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/15 "2022-05-18T13:10:37Z")
What is the error you are seeing when you are trying to deploy an EI backed endpoint? ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [May 24, 2022, 4:17am UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/16 "2022-05-24T04:17:49Z")
> [@schopra](#): > > `huggingface_model` @philschmid As requested, please find the error details below. ValueError: Unsupported image scope: eia. You may need to upgrade your SDK version (pip install -U sagemaker) for newer image scopes. Supported image scope(s): training, inference. Sagemaker SDK version = 2.87.0 huggingface\_model = HuggingFaceModel( model\_data=model\_data, # path to your trained SageMaker model role=get\_execution\_role(), # IAM role with permissions to create an endpoint entry\_point=‘deploy\_ei.py’, transformers\_version=“4.12.3”, # Transformers version used pytorch\_version=“1.9.1”, # PyTorch version used py\_version=‘py38’, # Python version used sagemaker\_session=sagemaker\_session ) ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [June 21, 2022, 12:50pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/17 "2022-06-21T12:50:00Z")
Hello @philschmid Any updates on this issue and EI (Elastic Inference) DLCs for inference? Thanks ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [June 22, 2022, 12:28pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/18 "2022-06-22T12:28:40Z")
Let me reach out to the AWS team again. I ll report back here as soon as I hear something. Out of curiosity why are you interested in EIA and not using Inferentia? ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [June 22, 2022, 3:17pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/19 "2022-06-22T15:17:34Z")
We are interested in cost effective solution and also interested in hosting multiple models in one container. But I think we can not host multiple models in one container behind one endpoint with both elastic inference and Inferentia but it’s possible with only cpu based instances. Thanks ---
**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\ **Post date:** [October 15, 2022, 2:20pm UTC](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736/20 "2022-10-15T14:20:44Z")
Hey @philschmid Any updates on this issue and EI (Elastic Inference) DLCs for inference? Thanks [Next page](https://discuss.huggingface.co/t/deploying-huggingface-sagemaker-models-with-elastic-inference/8736.md?page=2)