# The effects of ion implantation damage to photonic crystal optomechanical
resonators in silicon

Cliona Shakespeare  Department of Physics and Nanoscience Center, University
of Jyväskylä, P.O. Box 35, FI-40014 University of Jyväskylä, Finland  Teemu
Loippo  Department of Physics and Nanoscience Center, University of Jyväskylä,
P.O. Box 35, FI-40014 University of Jyväskylä, Finland  Henri Lyyra
Department of Physics and Nanoscience Center, University of Jyväskylä, P.O.
Box 35, FI-40014 University of Jyväskylä, Finland  Juha T. Muhonen
<EMAIL_ADDRESS>Department of Physics and Nanoscience Center, University
of Jyväskylä, P.O. Box 35, FI-40014 University of Jyväskylä, Finland

###### Abstract

Optomechanical resonators were fabricated on a silicon-on-insulator (SOI)
substrate that had been implanted with phosphorus donors. The resonators’
mechanical and optical properties were then measured (at 6 kelvin and room
temperature) before and after the substrate was annealed. All measured
resonators survived the annealing and their mechanical linewidths decreased
while their optical and mechanical frequencies increased. This is consistent
with crystal lattice damage from the ion implantation causing the optical and
mechanical properties to degrade and then subsequently being repaired by the
annealing. We explain these effects qualitatively with changes in the silicon
crystal lattice structure. We also report on some unexplained features in the
pre-anneal samples. In addition, we report partial fabrication of
optomechanical resonators with neon ion milling.

Ion implantation is a widely-used technology for doping of semiconductors with
applications varying from microchips [1, 2], to solar panels [3], to quantum
computing [4]. As ion implantation is a somewhat destructive method, basically
bombarding the substrate with high-energy ions, the process inevitably damages
the crystal structure of the substrate. This problem is especially acute with
silicon, as it is easily damaged and can become amorphous [5].

The implantation damage is usually healed with a post-implantation anneal [6]
that both allows the crystal to repair itself and activates any substitutional
donor atoms as they can then take their place in the crystal structure. Here
we study the effects of the implantation damage and subsequent annealing to
the mechanical and optical properties of suspended photonic crystal nanobeams.

The degradation of the optical properties of silicon from ion implantation is
of importance to any devices combining ion implantation and photonics in
silicon, a field that is growing in importance as single-photon emitters in
silicon are being actively investigated [7, 8, 9, 10], and could also affect
other silicon photonics devices such as vertical-cavity surface-emitting
lasers [11]. Similarly, the degradation of mechanical properties affects any
studies combining implanted ions and mechanical systems (such as hybrid
structures combining donor spin qubits [12, 13] with phonon buses [14]).

The resonators we study, as shown in Fig. 1, are sliced photonic crystal
nanobeams, the design of which was presented in Ref. 15. Similar structures
were also used in later studies [16, 17]. The photonic crystal in the center
of the nanobeam supports different optical modes than the photonic crystal on
the sides which then work as Bragg reflectors for a targeted cavity mode. Note
that here we do not have a tapering region between the mirrors and the cavity,
although it has been shown to decrease the optical linewidth considerably
[16]. The cavity optical mode has an electric field maximum at the center of
the silicon beam, which is also the location of the highest mechanical
displacement (for the fundamental mechanical mode). The small vacuum gap
between the two beam halves leads to strong field confinement in the gap, and
a large frequency dependence of the optical mode on the gap size. This gives
strong optomechanical coupling for the mechanical mode where the two beam
halves move asymmetrically in-plane (breathing mode).

Figure 1: a) Scanning electron microscope (SEM) image of an example nanobeam.
Gray parts are free hanging silicon and the dark parts show the substrate 3 µm
below. The optical cavity is defined in the center of the silicon nanobeam
where the photonic crystal lattice constant is smaller. The two beam halves
have been separated by a small gap. Inset: zoom to the gap between nanobeam
teeth showing the roughly 20 nm separation before suspension that was defined
with neon ion milling. b) Schematic of the interferometer used for
measurements; QWP refers to a quarter-wave-plate, HWP to a half-wave-plate,
PBS to a polarizing beam splitter and RSA to a realtime spectrum analyzer.

The samples were fabricated on a Silicon-On-Insulator (SOI) wafer with 220 nm
device layer and 3 µm buried oxide. The device layer was implanted with
phosphorus ions. The ion implantation [18] aimed for a constant donor density
of $10^{17}$ ions/cm3 in the device layer and was done in three steps: 110 keV
energy with a dose of $1.5\times 10^{12}$ ions/cm2, 40 keV with $5\times
10^{11}$ ions/cm2 and 20 keV with $1.5\times 10^{11}$ ions/cm2. The wafer was
not annealed after implantation, leaving any potential lattice damage intact.

The wafer was then diced and optomechanical resonators were fabricated on
chips. The chips were spin coated first with Microchem 950 k PMMA A4 resist,
which was then baked for 3 min on a hotplate at 160∘C, then with Allresist AR-
PC 5090.02 (Electra 92), which was baked for 1 min at 80∘C on a hotplate. The
resist was patterned with a Raith eLine electron beam lithography tool. The
PMMA was developed for 20 s in a 2:1 mixture of IPA and MIBK. Before
development, the Electra 92 resist was washed off. The etching was done with
an Oxford Plasmalab 80 reactive ion etcher with 80 sccm SF6 and 20 sccm O2 at
30 mTorr at $-$100∘C using liquid nitrogen cooling.

The gap between the beam halves was afterwards milled through with 25 kV neon
ions in a Zeiss Orion NanoFab helium ion microscope (HIM). Simulations of the
neon milling with SRIM [19] (28 keV ion energy, 3-4 pA ion current, 7.5 nC/µm2
area dose) show a lateral ion path of up to 30 nm, so there should be only a
moderate amount of neon ions in the silicon nanobeams from the milling.
Finally the optomechanical structures were released by wet etching the oxide
layer with 48 % HF for 4 min. An example of a released structure is shown in
Fig. 1 (a).

A picture of the milled gap between the beam halves is shown in the inset of
Fig. 1 (a) where the high-resolution enabled by neon milling is demonstrated
as the gap is only 20 nm wide. This would be beneficial for optomechanical
applications as the optomechanical coupling strength in these devices grows
with decreasing gap size. However, SEM images taken after the HF release show
an opening of roughly 50 nm. This could be due to the beam halves bending
outwards from stress or charging but requires further study.

The samples were placed inside a Montana Instruments C2 s50 cryostat and
measured with a balanced homodyne interferometer (schematic in Fig. 1(b))
using near infrared wavelengths. The optical cavity was coupled into from
above with a free space laser beam and the reflected light was interfered with
a reference laser beam. The interference signal as a function of laser
wavelength was measured with a subtracting detector and a spectrum analyzer.
Lorentzian line shapes were fitted to the brightest mechanical peak visible in
each spectrum (using the Python package SciPy [20]) and the optical and
mechanical resonance frequencies and linewidths were extracted from these
fits. As we do not lock the phase of the interferometer but average over all
the phases, our mechanical peaks split with the frequency of the piezo sweep
(c. 100 Hz) and the effective optical linewidth is modified, both of which we
account for in our fitting procedures.

After measuring the resonators at both room temperature and 6 K, the samples
were taken out and annealed in a tube furnace for 30 min at 600∘C, 15 s at
1000∘C, and 15 min at 500∘C. All annealing was done in an argon atmosphere
with 1 atm pressure. The chip was left to cool overnight, after which the chip
was loaded into the cryostat and the measurements at room temperature and 6 K
repeated.

The wavelength dependent spectrograms of the interferometer signal from one of
the resonators are shown in Fig. 2. The mechanical response is visible when
the laser wavelength is close to the optical cavity resonance frequency. An
unexpected phenomenon was an oscillation of the mechanical resonance frequency
as a function of the laser wavelength in the pre-anneal samples at 6 K, as
shown in Fig. 2(a). This effect was present in both resonators though much
more noticeable with resonator 2 and disappeared after annealing. The
oscillation was consistent between consecutive measurements (meaning the same
wavelength would produce same response) but varied when the laser spot
position (with regards to the silicon nanobeam) was moved. Changing the laser
power by a factor of 5 had no significant effect on the phenomenon, implying
it is not a thermal effect. We also note that these pre-anneal spectrograms at
6 K seem like they are a combination of two different peaks and we have fitted
them accordingly for the analysis below.

Figure 2: Resonator 2 spectrogram measured at 6 K before (a) and after (b)
annealing as a function of the laser wavelength. The colorbar shows the power
of the interferometer signal in dB. Crosscuts of the spectra (on a linear
scale) are shown in the insets with Lorentzian fits (dashed line). The post-
anneal sample exhibits much clearer Lorentzian resonance. The reference levels
for the signal are arbitrary.

After annealing the mechanical resonance frequencies increased consistently by
10-20 % for both samples at both temperatures, as pictured in Fig. 3, with
pre-anneal frequencies lying in the range of 7-8 MHz. An increase in resonance
frequency implies a stiffer resonator, i.e. a higher Young’s modulus. While
the Young’s modulus for silicon depends on crystal orientation [21], it is
higher for crystalline silicon in the fabrication direction of our device —
[110] — than it is for polycrystalline silicon, meaning that the shift in
resonance frequency can be at least qualitatively explained by the lattice
structure being damaged by the ion implantation, making it in effect more
polycrystalline-like and then being reconstructed by the annealing.

Figure 3: Plots of the resonators’ mechanical resonance frequencies and
linewidths before and after the annealing at room temperature (a) and 6 K (b).
The values before and after annealing have been connected with blue (Resonator
1) and green (Resonator 2) arrows for clarity.

To study the effect further, we made COMSOL simulations of the beams’
mechanical resonance frequencies for both polycrystalline Si and Si with the
same crystal direction as our sample using Young’s moduli from Ref. 21. The
simulated resonance frequency for polycrystalline silicon nanobeams, 6.7 MHz,
was lower than the 6.9 MHz of crystalline silicon, as expected. However, the
measured shifts are consistently higher than this effect alone would predict.

The mechanical linewidths for both resonators decreased after the anneal.
Interestingly all measured modes (including unshown less bright modes) ended
up with a linewidth of $0.2-0.3$ kHz at cryogenic temperatures after the
anneal, possibly indicating a limit where some mechanism not related to
temperature or crystal damage is determining the linewidth (such as clamping
losses). Both linewidths decreased by roughly a factor of 5. In room
temperature measurements resonator 1’s linewidth improved by a factor of 2
whereas resonator 2 had an improvement by factor of 6. The trend is consistent
with the lattice structure healing from implantation damage during the
annealing, as lattice defects can cause mechanical damping [22].

A similar plot for the optical properties is shown in Fig. 4. The optical
cavity resonance frequency and cavity linewidth can be extracted from the
strength of the optomechanical signal. The optical resonance frequencies
increased by $\sim$1% during the anneal. Here there are two effects that might
be playing a role: i) a decrease in the effective refractive index or ii)
change in the equilibrium gap size between the beams due to mechanical
reasons. We unfortunately cannot completely untangle these effects. The
refractive index of polycrystalline silicon is higher than that of single-
crystal silicon [23]. Thus recrystallization of the silicon lattice could lead
to a lowering of refractive index and a resonance shift towards higher optical
frequencies in accordance with our results. We also imaged the sample with SEM
before and after annealing and saw no visible change in the gap size. Note
that the doping concentration should not affect the effective refractive index
at these wavelengths and doping levels [24].

Figure 4: Plots of the resonators’ optical resonance frequencies and
linewidths before and after the annealing at room temperature (a) and 6 K (b).
The values before and after annealing have been connected with blue (Resonator
1) and green (Resonator 2) arrows for clarity.

For the optical linewidth we did not recover any consistent trend. We note
that there could be two competing factors here. Silicon crystal lattice damage
can be expected to cause photon losses to increase, and polycrystalline
silicon also has a higher coefficient of absorption at these wavelengths [25],
which means that annealing should decrease the linewidth. On the other hand,
doping activation during annealing would be expected to cause photon losses to
increase. We do note that after annealing there seems to be no significant
temperature dependence in the optical linewidth.

In conclusion, we have shown that post-implantation annealing has a
significant effect on the optical and mechanical properties of silicon
photonic crystal nanobeams. The exact mechanisms of the effects we
demonstrated still require further study but can qualitatively be explained by
implantation damage making the silicon more polycrystalline-like and annealing
then recovering the single-crystalline structure. These findings will have
relevance to any nanophotonic device that combines ion implantation with
photonic crystal structures, particularly the emerging field of spin-photonics
in silicon as well as to any structures combining ion implantation and
mechanical devices in silicon. The oscillation of the mechanical resonance
frequency as a function of the laser wavelength in the pre-anneal samples is
an intriguing phenomenon that could bear further investigation.

###### Acknowledgements.

We acknowledge Luke Antwis and Roger Webb for help with the ion beam
implantations. This project has received funding from the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation
programme (grant agreement No 852428) and by Academy of Finland Grant No
321416.

## References

  * Poate and Saadatmand [2002] J. M. Poate and K. Saadatmand, Review of Scientific Instruments 73, 868 (2002).
  * Rubin and Poate [2003] L. Rubin and J. Poate, Industrial Physicist 9, 12 (2003).
  * Rohatgi  _et al._ [2012] A. Rohatgi, D. L. Meier, B. McPherson, Y.-W. Ok, A. D. Upadhyaya, J.-H. Lai, and F. Zimbardi, Energy Procedia 15, 10 (2012).
  * van Donkelaar  _et al._ [2015] J. van Donkelaar, C. Yang, A. D. C. Alves, J. C. McCallum, C. Hougaard, B. C. Johnson, F. E. Hudson, A. S. Dzurak, A. Morello, D. Spemann, and D. N. Jamieson, Journal of Physics: Condensed Matter 27, 154204 (2015).
  * Williams [1998] J. S. Williams, Materials Science and Engineering: A 253, 8 (1998).
  * Ziegler [1985] J. F. Ziegler, Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms 6, 270 (1985).
  * Chen  _et al._ [2020] S. Chen, M. Raha, C. M. Phenicie, S. Ourari, and J. D. Thompson, Science 370, 592 (2020).
  * Bergeron  _et al._ [2020] L. Bergeron, C. Chartrand, A. T. K. Kurkjian, K. J. Morse, H. Riemann, N. V. Abrosimov, P. Becker, H.-J. Pohl, M. L. W. Thewalt, and S. Simmons, PRX Quantum 1, 020301 (2020).
  * Weiss  _et al._ [2021] L. Weiss, A. Gritsch, B. Merkel, and A. Reiserer, Optica 8, 40 (2021).
  * Durand  _et al._ [2021] A. Durand, Y. Baron, W. Redjem, T. Herzig, A. Benali, S. Pezzagna, J. Meijer, A. Y. Kuznetsov, J.-M. Gérard, I. Robert-Philip, M. Abbarchi, V. Jacques, G. Cassabois, and A. Dréau, Phys. Rev. Lett. 126, 083602 (2021).
  * Siriani and Choquette [2010] D. Siriani and K. Choquette, Electronics Letters 46, 712 (2010).
  * Pla  _et al._ [2012] J. J. Pla, K. Y. Tan, J. P. Dehollain, W. H. Lim, J. J. L. Morton, D. N. Jamieson, A. S. Dzurak, and A. Morello, Nature 489, 541 (2012).
  * Muhonen  _et al._ [2014] J. T. Muhonen, J. P. Dehollain, A. Laucht, F. E. Hudson, R. Kalra, T. Sekiguchi, K. M. Itoh, D. N. Jamieson, J. C. McCallum, A. S. Dzurak, and A. Morello, Nature Nanotechnology 9, 986 (2014).
  * Patel  _et al._ [2018] R. N. Patel, Z. Wang, W. Jiang, C. J. Sarabalis, J. T. Hill, and A. H. Safavi-Naeini, Phys. Rev. Lett. 121, 040501 (2018).
  * Leijssen and Verhagen [2015] R. Leijssen and E. Verhagen, Scientific Reports 5, 15974 (2015).
  * Leijssen  _et al._ [2017] R. Leijssen, G. R. La Gala, L. Freisem, J. T. Muhonen, and E. Verhagen, Nat. Commun. 8, ncomms16024 (2017).
  * Muhonen  _et al._ [2019] J. T. Muhonen, G. R. La Gala, R. Leijssen, and E. Verhagen, Phys. Rev. Lett. 123, 113601 (2019).
  * [18] Implantation was carried out in the Ion Beam Centre of University of Surrey.
  * Ziegler, Ziegler, and Biersack [2010] J. F. Ziegler, M. D. Ziegler, and J. P. Biersack, Nuclear Instruments and Methods in Physics Research B 268, 1818 (2010).
  * Virtanen  _et al._ [2020] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, Nature Methods 17, 261 (2020).
  * Hopcroft, Nix, and Kenny [2010] M. A. Hopcroft, W. D. Nix, and T. W. Kenny, Journal of Microelectromechanical Systems 19, 229 (2010).
  * Shin  _et al._ [2015] J. Shin, K. Choi, S. Shiko, H. Choi, and D. Bae, Composites Part B: Engineering 77, 194 (2015).
  * Jones and Wesolowski [1984] R. E. Jones and S. P. Wesolowski, Journal of Applied Physics 56, 1701 (1984).
  * Auslender and Hava [2017] M. Auslender and S. Hava, “Single-crystal silicon: Electrical and optical properties,” in  _Springer Handbook of Electronic and Photonic Materials_, edited by S. Kasap and P. Capper (Springer International Publishing, Cham, 2017) pp. 1–1.
  * Harbeke  _et al._ [1984] G. Harbeke, L. Krausbauer, E. F. Steigmeier, A. E. Widmer, H. F. Kappert, and G. Neugebauer, Journal of The Electrochemical Society 131, 675 (1984).


# Stable Code Technical Report

Nikhil Pinnaparaju Reshinth Adithyan Duy Phung Jonathan Tow
James Baicoianu Ashish Datta Maksym Zhuravinskyi
Dakota Mahan Marco Bellagente Carlos Riquelme Nathan Cooper

Stability AI Language Team Correspondance to: {nikhil.pinnaparaju, reshinth,
<EMAIL_ADDRESS>
###### Abstract

We introduce Stable Code, the first in our new-generation of code language
models series, which serves as a general-purpose base code language model
targeting code completion, reasoning, math, and other software engineering-
based tasks. Additionally, we introduce an instruction variant named Stable
Code Instruct that allows conversing with the model in a natural chat
interface for performing question-answering and instruction-based tasks. In
this technical report, we detail the data and training procedure leading to
both models. Their weights are available via Hugging Face for anyone to
download and use 111https://huggingface.co/stabilityai/stable-
code-3b222https://huggingface.co/stabilityai/stable-code-instruct-3b. This
report contains thorough evaluations of the models, including multilingual
programming benchmarks, and the MT benchmark focusing on multi-turn dialogues.
At the time of its release, Stable Code is the state-of-the-art open model
under 3B parameters and even performs comparably to larger models of sizes 7
billion and 15 billion parameters on the popular Multi-PL benchmark. Stable
Code Instruct also exhibits state-of-the-art performance on the MT-Bench
coding tasks and on Multi-PL completion compared to other instruction tuned
models. Given its appealing small size, we also provide throughput
measurements on a number of edge devices. In addition, we open source several
quantized checkpoints and provide their performance metrics compared to the
original model.

##  1 Introduction

The application of Machine Learning in programming languages has been
prolific: from code understanding to code representation or code completion.
Earlier work focused on exploiting the underlying deep semantic structure of
programming languages like in Code2Vec [4], Code2Seq [3], Graph Representation
Learning for Code [2]. The above architectures are tailor-made for the native
structures of Abstract Syntax Trees (AST) / Data Flow Graphs (DFG), and have a
significant limitation: they can only be applied for tasks that involve
completely executable code. For instance, to generate DFG as outlined by
Allamanis et al.[2], the program needs to be complete. Works such as Austin et
al., [5] have shown how transformers-based models can be used like natural
language for code at the lexical (text) level. Since then, language models
have been widely used to model code on a variety of tasks. The extensible
training dynamics of language models offer improved reasoning and
understanding ability [41] often captured via their application in
productivity tools for software engineers such as code completion [20, 23],
question answering [18, 20], and debugging plugins [15, 23]. Models like these
are executed every few seconds, especially in the case of code completion.
Accordingly, strong models that can run on consumer devices are preferred, as
avoiding network latency makes a difference and addresses various
discrepancies with respect to gated APIs. In this work, we introduce Stable
Code and Stable Code Instruct, small and fast code language models that
achieves strong performance on popular benchmarks while matching significantly
larger models.

The report is structured as follows. Section 2 describes the training data in
detail, and the architectural choices are provided in Section 3. Section 4
covers the training procedure applied to the base Stable Code model, while
Section 5 focuses on instruction tuning. In Section 6 we share the performance
of these models and Section 7 is centered around throughput and quantization.
Finally, Section 8 concludes.

Dataset |  Sampling Weight |  Num Tokens Sampled |  Epochs |  Category
---|---|---|---|---
StarCoder C |  0.0924 |  122,202,657,912.00 |  6.0 |  Code
StarCoder CPP |  0.0734 |  97,032,316,152.00 |  6.0 |  Code
StarCoder Java |  0.1029 |  136,010,698,326.00 |  6.0 |  Code
StarCoder Javascript |  0.0858 |  113,469,977,934.00 |  6.0 |  Code
StarCoder CSS |  0.0146 |  19,285,266,328.00 |  4.0 |  Code
StarCoder Go |  0.0258 |  34,092,166,492.00 |  4.0 |  Code
StarCoder HTML |  0.0298 |  39,354,336,188.00 |  4.0 |  Code
StarCoder Ruby |  0.0061 |  8,011,730,332.00 |  4.0 |  Code
StarCoder Rust |  0.0122 |  16,131,445,656.00 |  6.0 |  Code
StarCoder Markdown |  0.1154 |  152,629,435,716.00 |  6.0 |  Code
StarCoder Shell |  0.0033 |  4,323,112,416.00 |  4.0 |  Code
StarCoder Php |  0.0764 |  100,958,420,706.00 |  6.0 |  Code
StarCoder Sql |  0.0247 |  32,645,285,202.00 |  6.0 |  Code
StarCoder R |  0.0003 |  415,957,896.00 |  4.0 |  Code
StarCoder Typescript |  0.0224 |  29,634,722,636.00 |  4.0 |  Code
StarCoder Python |  0.1067 |  141,067,150,184.00 |  8.0 |  Code
StarCoder Jupyter |  0.0060 |  7,941,540,044.00 |  4.0 |  Code
StarCoder Restructured Text |  0.0032 |  4,179,202,492.00 |  4.0 |  Code
Github Issues |  0.0231 |  46,302,993,820 |  2.5 |  Technical
Github Diffs |  0.0019 |  3,817,060,582 |  2.0 |  Technical
StackExchange |  0.0019 |  3,817,060,582 |  2.0 |  Technical
Synthetic |  0.0006 |  819,864,748.00 |  3.0 |  Technical
Proof Pile |  0.0384 |  50,780,637,096 |  1.0 |  Math
Meta Math QA |  0.0003 |  83,663,501 |  4.0 |  Math
Arxiv |  0.0213 |  28,097,511,912.00 |  1.0 |  Web
Refined Web |  0.0220 |  29,114,185,066.13 |  0.5 |  Web
Total |  1 |  1,322,090,182,830.13 |  |  -
Table 1: The complete Stable Code 3B training set with sampling weights. The
tokens count refers to the NeoX tokenizer introduced in Sec. 3.1

##  2 Training Data

###  2.1 Pretraining Dataset

To construct the pre-training dataset for Stable Code, we collected a diverse
array of publicly accessible, large-scale data sources. These sources
encompass a wide spectrum of code repositories, extensive collections of
technical documents (example: readthedocs), mathematically focused texts, and
comprehensive web datasets. The primary objective of this initial pretraining
phase is to learn a rich internal representation that extends beyond mere code
comprehension. Our aim was to significantly enhance the model’s capabilities
in mathematical understanding, logical reasoning, and processing of complex
technical texts surrounding software development. The rationale behind
selecting such a varied dataset mix is to develop a language model well suited
to perform a wide range of software engineering tasks, not only those directly
related to programming like code completion.

Furthermore, our training data also contains general text datasets in order to
provide the model with a broader linguistic knowledge and context. We hope
this enables the model to address a wider range of queries and tasks in a
conversational manner. In Table 1, we provide the data sources, epochs,
categories, and sampling weights of the datasets used to set up the
pretraining corpus. We use an 80:20 split distribution of code and natural
language data, respectively, with the contributions from individual components
detailed in the table (see Table 2.1 for references).

References for main training datasets. |
---|---
Dataset | Reference
Dataset | Reference
Continued on next page
StarCoder Data |  [26]
Github Issues |  [25]
Github Diffs |  [28]
Stackexchange |  [16]
Arxiv |  [16]
Synthetic Dataset | Sec 2.1.1
Proof Pile Math |  [6]
Meta Math QA |  [42]
Refined Web |  [30]

####  2.1.1 Synthetic Dataset

We also introduce a small synthetic dataset into our pre-training corpus. The
data is synthetically generated from the seed prompts of the
CodeAlpaca333Subset found here -
https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K dataset, which is
comprised of 174,000 prompts. To augment the diversity and difficulty
presented in the code-alpaca prompts, we employed the "Evol-Instruct" method
as introduced by Xu et al., [40] wherein we ask a language model (in this
case, we use WizardLM [40]) to increase the complexity of the given seed
prompt in a step-by-step fashion. By applying strategies focused on breadth,
reasoning, deepening, and complexity, we were able to enrich our collection
with an additional 100,000 prompts. We leverage the DeepSeek Coder 34B model
[21] to generate synthetic outputs for the newly developed "Evol-Instruct"
prompts. We believe that introducing this synthetic data early during the
pretraining phase helped the model respond better to natural language text
based on an ablation experiment we conducted.

###  2.2 Long Context Dataset

Building upon the initial pre-training phase, we composed an additional stage
of training that specifically targets the model’s capability to process and
comprehend long sequences. Having a longer context length is useful for coding
models due to the usual inter-dependence of multiple files within a
repository. We specifically chose 16,384 as the context length for our long
context dataset after determining the median and mean number of tokens in a
software repository to be $\approx 12k$ and $\approx 18k$ tokens,
respectively. This continued training stage focused on a curated selection of
programming languages, all sourced from The Starcoder dataset [26], a filtered
version of the Stack which is a large collection of high quality and
permissively licensed coding data [25]. The languages selected for this phase
were based on the Stack Overflow Developer Survey 2022 [35]. In particular, we
selected Python, C, C++, Go, Java, and JavaScript.

To create this long context dataset, we took files written in these languages
within a repository and combined them, inserting a special <repo_continuation>
token between each file to maintain separation while preserving the flow of
content. To circumvent any potential biases that might arise from a fixed
ordering of files—a factor that could inadvertently teach the model an
unintended sequence or hierarchy—we employed a randomized strategy. For each
repository, we generated not one, but two distinct orderings of the
concatenated files. The statistics are outlined in Table 2.2.  Data
distribution of files before and after concatenation Statistic File Level Long
Context Statistic File Level Long Context Continued on next page Percentage of
Rows >= 4096 5 100 Median 470 12834 Max Length 326833 1020069 Min Length 3
6008 Mean 1195 18000

##  3 Model Architecture

Stable Code is built on top of Stable LM 3B [38], which is a state-of-the-art
LLM for natural language in English at the 3 billion parameter scale. The
model is a causal decoder-only transformer similar in design to the LLaMA
architecture [37]. Table 3 shows some of the key architectural details. In
particular, the main differences with respect to LLaMA are the following:

  * •

Position Embeddings. Rotary Position Embeddings [36] applied to the first
$25\%$ of head embedding dimensions for improved throughput following [13].

  * •

Normalization. LayerNorm [7] with learned bias terms as opposed to RMSNorm
[43].

  * •

Biases. We remove all bias terms from the feed-forward networks and multi-head
self-attention layers, except for the biases of the key, query, and value
projections [8].

###  3.1 Tokenizer

We use the same tokenizer as the Stable LM 3B model, which is based on the BPE
tokenizer from Black et al.,[13] with a vocabulary size of 50,257. We also
added the special tokens from the StarCoder models that include tokens for
indicating the name of the file, the number of stars for the repository, Fill-
in-Middle (FIM), etc. For our long context training stage, we added a special
token to indicate when two concatenated files belong to the same repository.

Parameters |  Hidden Size |  Layers |  Heads |  Sequence Length
---|---|---|---|---
2,795,443,200 |  2560 |  32 |  32 |  16384

Table 2: Stable Code 3B model architecture.

Precision |  Micro Batch Size |  Gradient Accumulation Steps |  Activation Checkpointing
---|---|---|---
BF16 |  4 |  1 |  enabled

Table 3: Stable Code 3B training configuration.

##  4 Training

In this section, we elaborate on the methodologies adopted for training the
Stable Code models. This detailed analysis covers the compute infrastructure,
setup configurations, and training techniques employed to optimize the model’s
learning process. We further delve into the use of the fill-in-the-middle
(FIM) training objective and conduct a series of experiments to evaluate the
effects of various initializations on the model’s performance.

###  4.1 Compute and Setup

Stable Code was trained on 32 Amazon P4d instances comprising 256 NVIDIA A100
(40GB HBM2) GPUs. The size of our model, together with ZeRO stage 1
distributed optimization [33], eliminates the need for model sharding. Still,
different triplets of micro-batch size, gradient accumulation steps, along
with activation checkpointing granularity lead to different speed metrics. We
employ a global batch size of $4,194,304$ tokens. With the setup in Table 3,
we achieve $\approx$222 TFLOPs/s per device, or $71.15\%$ model flops
utilization (MFU).

###  4.2 Multi Stage Training

We employed a staged training approach that has been popular in other strong
code language models such as the CodeGen [29], Stable Code Alpha [1],
CodeLLaMA [34], and DeepSeekCoder [21] models as shown in Figure 1.

Figure 1: Staged approach to training Stable Code 3B and Stable Code Instruct
3B.

We train Stable Code to predict the next token following standard
autoregressive sequence modeling [31]. We initialize our model from the Stable
LM 3B checkpoint using the same base context length, $4096$ for the first
stage of training with the data mix detailed in Section 2. This is followed by
a continued pretraining stage as shown in Figure 1.

Training is performed in BFloat16 mixed precision while keeping all-reduce
operations in FP32. We use a standard AdamW optimizer with the following
hyperparameters:
$\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1e-6,\lambda(\text{weight decay})=0.1$.
We start with a learning rate of 3.2e-4, set a minimum learning rate of
3.2e-5, and use a cosine decay learning rate schedule, shown in Fig 2

Figure 2: Stable Code 3B Loss and Learning Rate Curves.

The long context finetuning stage uses a peak learning rate of 2e-5 with a
minimum of 1.28e-5 with the same betas, epsilon, and decay rate. We also
switch the rotary embedding base to $1,000,000$ in line with CodeLLama [34]
and Stable Code Alpha [1]. The long context finetuning stage is performed for
another 20,000 steps, i.e. an additional $\approx$ 300 billion tokens.

###  4.3 Model Initialization

As with other types of models, code models mostly follow one of two main
training recipes: models trained from scratch with code and related texts
(e.g., CodeGen [29], Stable Code Alpha [1], Deepseek Coder [21]) and models
leveraging continued pretraining from a foundational language model (e.g.,
works akin to CodeLLaMA [34]). As possibly expected, we observe in Figure 3
that models initialized from the weights of a pretrained language model (say
Stable LM 3B [38]) tend to outperform models trained from scratch. This
observation aligns with the hypothesis that a positive cross-transfer between
natural language and code contributes to enhanced model capabilities,
supported by insights from the "Naturalness of Software" [19] literature.

The Stable LM 3B base model was initially pre-trained on a dataset comprising
100 billion code tokens (in addition to 3.9 Trillion non-code tokens), which
forms the baseline for all line charts depicted in green in Figure 3. This
token count is consistently incorporated into the total token count for all
subsequent training phases, ensuring an equi-code-token comparison across
models. Figure 3 shows us that at no point the line charts in blue cross the
performance of those in green, suggesting that training on natural language
tokens helps improve code capabilities of language models.

Figure 3: Code Performance Comparison of Stable Code 3B Scratch and Stable LM
3B Initializations

###  4.4 Fill in the Middle (FIM) Training

One of the core assumptions for natural language models training is the left-
to-right causal ordering of tokens (note there are some exceptions, like
Arabic). In the case of code, this assumption does not always hold (e.g.,
function calls and function declarations can be in an arbitrary order for many
programming languages). To address this, we use the "Fill in the Middle"
objective [10]. The approach involves splitting a document randomly into three
segments: prefix, middle, and suffix, and then moving the middle segment to
the end of the document. After the rearrangement the same autoregressive
training process is followed.

This rearrangement allows for conditioning on structural patterns besides the
prefix-only form used in traditional causal language modeling. The augmented
data can then be further organized into two modes: “Suffix-Prefix-Middle”
(“SPM”) and “Prefix-Suffix-Middle” (“PSM”). Following [10], we apply FIM at
the character-level using a rate of 50% and choose between “SPM” and “PSM”
modes with uniform probability.

FIM was applied during both stages of pretraining. In order to account for FIM
in the long context training phase, we made sure to only allow FIM to be
applied within the confines of individual files to avoid introducing
unrealistic scenarios into the training objective.

##  5 Fine-tuning and Alignment

Following pre-training, we further improve our model’s conversational skills
via a fine-tuning stage that consists of supervised fine-tuning (SFT) and
direct preference optimization (DPO) [32].

We use the following datasets that are publicly available on Hugging Face for
SFT finetuning: OpenHermes
444https://huggingface.co/datasets/teknium/OpenHermes-2.5, Code Feedback
555https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction,
CodeAlpaca 666https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K.
Together they provide around $500,000$ training samples after performing an
exact match deduplication. We train our SFT models for three epochs using a
cosine learning rate scheduler. A warm-up phase of $10\%$ of the training
duration is employed before reaching the peak learning rate of 5e-5. We set
the global batch size to 512 sequences and pack inputs into sequences of up to
4096 tokens in length.

After SFT, we apply Direct Preference Optimization (DPO) [31], a technique
that has been key to the success of recent high-performing models like
Zephyr-7B [39], Neural-Chat-7B, and Tulu-2-DPO-70B [24], Stable LM 2 1.6B
Zephyr [11]. In this case, we curated a dataset of approximately 7,000
samples, leveraging data from UltraFeedback [17], Distilabel Capybara DPO-7k
Binarized 777https://huggingface.co/datasets/argilla/distilabel-capybara-
dpo-7k-binarized and only kept samples related to code by filtering using an
LLM based approach. Additionally, to improve the model safety we also included
the Helpful and Harmless RLFH dataset from Bai et al.[9]

Moreover, we added the harmless subset from the HH-Anthropic dataset
888https://github.com/anthropics/hh-rlhf/. We curated a dataset comprising
instances that received safety ratings of 0 or 1, indicating content that is
critical for training models to avoid generating or perpetuating harmful
outputs. This process resulted in the compilation of approximately 15,000
high-relevance safety-related data points.

We incorporate a DPO step that adheres to the Zephyr recipe, as outlined by
Tunstall et al [39]. The tuning of the $\beta$ hyperparameter, which is set to
$0.01$, following best practices for ensuring model stability and convergence.
Furthermore, we employ RMSProp as our optimization algorithm, the initial
phase of DPO training involves increasing the learning rate to a peak value of
5e-7, with $10\%$ warm-up step of the total training duration.

##  6 Results

Throughout this section, we showcase a comparative analysis of the Stable Code
and Stable Code Instruct models, with a range of selected baselines and models
previously recognized as state-of-the-art within the same computational scale
of 3B parameters. We also look at larger code language models, illustrating
the diverse competitive landscape our model navigates.

###  6.1 Code Completion Benchmarks

####  6.1.1 Stable Code Base

The primary metric for this comparison is the model’s performance on code
completion tasks, a fundamental evaluation criterion given its direct
relevance to the practical utility of code language models. We use the Multi-
PL benchmark proposed by Cassano et al., [14] to evaluate the models.

Performance of different base code LLMs with size 3B params and under on Multi-PL. |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
Model | Size | Avg |  Python |  C++ |  JavaScript |  Java |  PHP |  Rust |
Stable Code | 3B | 29.1 | 32.4 | 30.9 | 32.1 | 32.1 | 24.2 | 23.0
StarCoder v2 | 3B | 30.9 | 28.9 | 32.3 | 36.5 | 31.2 | 29.9 | 25.6 |
DeepSeek Coder | 1.3B | 26.1 | 28.6 | 29.2 | 28.7 | 29.0 | 23.6 | 18.5 |
Wizard Coder | 3B | 25.7 | 31.2 | 25.6 | 26.2 | 25.8 | 25.3 | 20.4 |
Replit Code V1.5 | 3B | 23.9 | 23.0 | 25.9 | 26.2 | 23.6 | 23.2 | 21.5 |
StarCoder | 3B | 19.9 | 21.6 | 19.8 | 21.5 | 20.5 | 19.0 | 16.9 |
Deci Coder | 1B | 10.8 | 19.1 | 6.8 | 18.4 | 16.7 | 2.1 | 1.7 |
Deepseek Coder | 6.7B | 43.0 | 45.9 | 48.7 | 46.1 | 43.0 | 38.6 | 35.4 |
Code Llama | 7B | 29.1 | 30.0 | 28.2 | 32.5 | 31.9 | 25.7 | 26.3 |
StarCoder | 15B | 29.0 | 33.6 | 31.6 | 30.8 | 30.2 | 26.1 | 21.8 |

Despite its relatively smaller size –less than 40% and 20% the number of
parameters of Code Llama [34] and StarCoder 15B [26], respectively– Stable
Code matches their performance on average across programming languages. Also,
Stable Code 3B achieves strong performance at the 3B scale, showing remarkable
capabilities in code completion tasks. The StarCoder v2 model [27] –which is a
newer model trained on significantly more data– outperforms Stable Code 3B
base on average. 999See the detailed and comprehensive work by the BigCode
team [27] for updated results around that model.

####  6.1.2 Stable Code Instruct

Similarly, we evaluate instruct-tuned variants of these models on the Multi-PL
benchmark with Stable Code Instruct in Table 6.1.2.

Performance of instruction tuned code LLMs with size around 3B params on Multi-PL. |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Model |  Size |  Avg |  Python |  C++ |  JS |  Java |  PHP |  Rust
|  |  |  |  |  |  |  |
Stable Code Instruct |  3B |  47.2 |  58.6 |  48.1 |  49.2 |  44.4 |  45.6 |  37.2
DeepSeek Coder Ins |  1.3B |  44.3 |  52.5 |  45.0 |  52.3 |  40.9 |  46.4 |  28.6
DeepSeek Coder Ins |  6.7B |  61.1 |  64.6 |  63.2 |  67.7 |  59.1 |  62.7 |  48.9
Codellama Instruct |  7B |  30.6 |  32.7 |  30.8 |  33.6 |  31.5 |  29.6 |  25.5

Stable Code Instruct offers a remarkably solid performance for its size.

###  6.2 Fill in the Middle

The capacity to utilize both preceding and succeeding context significantly
benefits non-instruct tuned code language models, enabling more precise and
context-aware completions. Therefore, we further extend our evaluation to the
specialized Fill in the Middle (FIM) code completion performance. In this
case, we utilize the StarCoder-FIM/SantaCoder-FIM evaluation tasks from the
BigCode Evaluation Harness [12], offering a nuanced assessment of each model’s
understanding and prediction capabilities in more complex coding contexts. We
focus on the best performing models from Table 6.1.1, and Table 6.2 shows the
FIM performance of such models.

FIM performance of different base code models with size at most 3B parameters. |  |  |  |  |
---|---|---|---|---|---
Model | Size |  Python |  JavaScript |  Java |
Stable Code | 3B | 59.1 | 73.4 | 64.1
StarCoder V2 | 3B | 59.0 | 72.8 | 74.9 |
DeepSeek Coder | 1.3B | 53.5 | 65.2 | 49.4 |
StarCoder | 3B | 50.3 | 61.4 | 56.1 |

###  6.3 Multi Turn Benchmark

We also evaluate instruct tuned models on the code subset of the challenging
Multi-turn benchmark (MT-Bench) [44]. Table 6.3 shows the results of coding
questions in MT-Bench.  MT-Bench Coding Question Score. Model Size MT-
Bench[Coding] Stable Code Instruct 3B 5.8 DeepSeek Coder 1.3B 4.6 DeepSeek
Coder 6.7B 6.9 CodeLlama Instruct 7B 3.6

###  6.4 SQL Performance

One important application for code language models are database query tasks.
In this domain, we compare the performance of Stable Code Instruct against
other popular instruction-tuned models and models specifically trained to
perform well in SQL. We use the benchmark created by Defog AI
101010https://defog.ai/blog/open-sourcing-sqleval/ to evaluate our models.
Evaluation of popular models and Stable Code Instruct 3B on SQL-Eval. Size Avg
Date Group By Order By Ratio Join Where Stable Code Instruct 3B 47.2 24.0 54.2
68.5 40.0 54.2 42.8 DeepSeek-Coder Instruct 1.3B 34.2 24.0 37.1 51.4 34.3 45.7
45.7 SQLCoder 7B 70.6 64.0 82.9 74.3 54.3 74.3 74.3

##  7 Inference

Speed and memory considerations are particularly important for code models.
Accordingly, in this section we briefly mention how to make Stable Code even
faster. In our release, we offer quantized weights for the Stable Code model
111111https://huggingface.co/stabilityai/stable-code-3b/tree/main, ensuring
compatibility with widely adopted inference libraries such as llama.cpp
121212https://github.com/ggerganov/llama.cpp and Apple MLX [22].

###  7.1 Quantization

We provide quantized model files in various formats to facilitate seamless
integration with a diverse range of inference engines. The offered
quantization files encompass the following models: GGUF Q5_K_M, GGUF Q6_K,
GGUF FP16, and MLX INT4.

###  7.2 Throughput

In Table 7.2, we present the throughput numbers obtained when running Stable
Code on consumer-grade devices and the corresponding system environments.
These results indicate a nearly two-fold increase in throughput when employing
lower precision. However, it is important to note that these data points serve
as a rough estimate and do not represent the outcome of comprehensive
benchmarking. Instead, they aim to offer users with a practical understanding
of potential performance improvements on typical hardware.

It is worth mentioning that implementing lower precision quantization may lead
to some reduction in model performance (maybe significant). Therefore, we
strongly advise researchers and developers to evaluate the actual impact on
their specific use cases in real-world scenarios before making decisions.

Throughput and power usage on various devices using different quantization frameworks. |  |  |  |
---|---|---|---|---
Framework |  CPU |  Precision |  Throughput (Tok/s) |  Power Consumption (W)
MLX |  M2 Pro Max |  FP16 |  23 |  18
MLX |  M2 Pro Max |  INT4 |  52 |  17
GGUF |  M2 Pro Max |  FP16 |  28 |  14
GGUF |  M2 Pro Max |  Q5_K_M |  53 |  23
GGUF |  M2 Pro Max |  Q6_K |  54 |  23

##  8 Conclusion

In this report, we introduce Stable Code and Stable Code Instruct, two compact
decoder-only language models targeting different software development use
cases. In the spirit of open science, we detail all datasets for pre-training
and alignment, and discuss our training and evaluation methodologies. We also
conduct extensive model evaluations and comparisons with other similarly-sized
models, demonstrating Stable Code and Stable Code Instruct’s remarkable
performance. Finally, we profile the model on common edge computing
architectures. We hope this report and the models we release contribute to the
improvement and further research on code language models and their use.

## References

  * [1] Reshinth Adithyan, Duy Phung, Nathan Cooper, Nikhil Pinnaparaju, and Christian Laforte.  Stable code complete alpha.
  * [2] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi.  Learning to represent programs with graphs.  ArXiv, abs/1711.00740, 2017.
  * [3] Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav.  code2seq: Generating sequences from structured representations of code.  ArXiv, abs/1808.01400, 2018.
  * [4] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav.  code2vec: learning distributed representations of code.  Proceedings of the ACM on Programming Languages, 3:1 – 29, 2018.
  * [5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  arXiv preprint arXiv:2108.07732, 2021.
  * [6] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck.  Llemma: An open language model for mathematics, 2023.
  * [7] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.
  * [8] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.  Qwen technical report, 2023.
  * [9] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
  * [10] Mohammad Bavarian, Heewoo Jun, Nikolas A. Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen.  Efficient training of language models to fill in the middle.  ArXiv, abs/2207.14255, 2022.
  * [11] Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, Meng Lee, Emad Mostaque, Michael Pieler, Nikhil Pinnaparju, Paulo Rocha, Harry Saini, Hannah Teufel, Niccolo Zanichelli, and Carlos Riquelme.  Stable lm 2 1.6b technical report, 2024.
  * [12] Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra.  A framework for the evaluation of code generation models.  https://github.com/bigcode-project/bigcode-evaluation-harness, 2022.
  * [13] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach.  Gpt-neox-20b: An open-source autoregressive language model, 2022.
  * [14] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda.  Multipl-e: A scalable and polyglot approach to benchmarking neural code generation.  IEEE Transactions on Software Engineering, 49(7):3675–3691, 2023.
  * [15] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou.  Teaching large language models to self-debug, 2023.
  * [16] Together Computer.  Redpajama: An open source recipe to reproduce llama training dataset, 2023.
  * [17] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback, 2023.
  * [18] Cursor.  Cursor: The ai-first code editor, 2024.
  * [19] Premkumar T. Devanbu.  On the naturalness of software.  2012 34th International Conference on Software Engineering (ICSE), pages 837–847, 2012.
  * [20] GitHub.  Github copilot: The world’s most widely adopted ai developer tool., 2024.
  * [21] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang.  Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024.
  * [22] Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert.  MLX: Efficient and flexible machine learning on apple silicon, 2023.
  * [23] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang.  Large language models for software engineering: A systematic literature review, 2023.
  * [24] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.  Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.
  * [25] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries.  The stack: 3 tb of permissively licensed source code.  Preprint, 2022.
  * [26] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  StarCoder: may the source be with you!, 2023.
  * [27] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al.  Starcoder 2 and the stack v2: The next generation.  arXiv preprint arXiv:2402.19173, 2024.
  * [28] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre.  Octopack: Instruction tuning code large language models.  arXiv preprint arXiv:2308.07124, 2023.
  * [29] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  Codegen: An open large language model for code with multi-turn program synthesis.  In International Conference on Learning Representations, 2022.
  * [30] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023.
  * [31] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training, 2018.
  * [32] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model, 2023.
  * [33] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models, 2020.
  * [34] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.  Code llama: Open foundation models for code, 2023.
  * [35] StackOverFlow.  Stackoverflow developer survey - 2022, 2022.
  * [36] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding, 2023.
  * [37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * [38] Jonathan Tow, Marco Bellagente, Dakota Mahan, and Carlos Riquelme.  Stablelm 3b 4e1t, 2023.
  * [39] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.  Zephyr: Direct distillation of lm alignment.  arXiv preprint arXiv:2310.16944, 2023.
  * [40] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  arXiv preprint arXiv:2304.12244, 2023.
  * [41] Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi Ren Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, Heng Ji, and Chengxiang Zhai.  If llm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents.
  * [42] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.  Metamath: Bootstrap your own mathematical questions for large language models.  arXiv preprint arXiv:2309.12284, 2023.
  * [43] Biao Zhang and Rico Sennrich.  Root mean square layer normalization, 2019.
  * [44] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.

##  Appendix A Examples

In this appendix, we briefly showcase some examples of Stable Code in action.
The context is shown in black whereas the suggestion completion is shown in
magenta.

⬇

1def bubbleSort(arr):

2 n = len(arr)

3

4 for i in range(n):

5 swapped = False

6

7 for j in range(0, n-i-1):

8 # if arr[j] > arr[j+1]:

9 arr[j], arr[j+1] = arr[j+1], arr[j]

10 swapped = True

11

12 if swapped == False:

13 break

Listing 1: Bubble Sort FIM Example.

⬇

1 return chat_model.invoke(messages);

2 }

3}

4

5export function fetchResponse(prompt: string, fileName: string, MODEL_NAME: string, API_KEY: string, modelType: string): Promise<AIMessage> | Promise<string> {

6 console.log("Fetching response", modelType);

7 switch (modelType) {

8 case "action":

9 return createChatModel(MODEL_NAME, API_KEY).invoke([new
HumanMessage(prompt)]);

10 case "completion":

11 return createCompletionModel(MODEL_NAME, API_KEY).invoke(prompt);

12 default:

13 throw new Error(#‘Unknown model type: ${modelType}‘);

14 }

15}

Listing 2: Typescript Completion Example.

⬇

1stage_3 = DiffusionPipeline.from_pretrained(

2 "stabilityai/stable-diffusion-x4-upscaler", **safety_modules,
torch_dtype=torch.float16

3)

4

5stage_3.enable_model_cpu_offload()

6

7prompt = ’a cute otter’

8generator = torch.manual_seed(1)

9

10prompt_embeds, negative_embeds = stage_1.encode_prompt(prompt)

11

12stage_1_output = stage_1(

13 prompt_embeds=#prompt_embeds, negative_embeds=negative_embeds,
generator=generator

14).images

15

16stage_2_output = stage_2(

17 image=stage_1_output,

18 prompt_embeds=prompt_embeds,

19 negative_prompt_embeds=negative_embeds,

20 generator=generator,

21 output_type="pt",

22).images

23

24stage_3_output = stage_3(prompt=prompt, image=stage_2_output,
noise_level=100, generator=generator).images

25make_image_grid([pt_to_pil(stage_1_output)[0], pt_to_pil(stage_2_output)[0],
stage_3_output[0]], rows=1, rows=3)

Listing 3: PyTorch Modeling Example.