* (70) Yin Y et al. 2015. Tunable permalloy-based films for magnonic devices. Phys. Rev. B 92, 024427. (doi: 10.1103/PhysRevB.92.024427)
  * (71) Brito WH, Aguiar MCO, Haule K, Kotliar G. 2016. Metal-Insulator Transition in VO2: A DFT+DMFT Perspective. Phys. Rev. Lett. 117, 056402. (doi: 10.1103/PhysRevLett.117.056402)
  * (72) Šmejkal L, Železný J, Sinova J, Jungwirth T. 2017. Electric Control of Dirac Quasiparticles by Spin-Orbit Torque in an Antiferromagnet. Phys. Rev. Lett. 118, 106402. (doi: 10.1103/PhysRevLett.118.106402)
  * (73) Yang H et al. 2021. Colossal angular magnetoresistance in the antiferromagnetic semiconductor EuTe2. Phys. Rev. B 104, 214419 (doi: 10.1103/PhysRevB.104.214419)
  * (74) Rijks TGSM, Coehoorn R, de Jong MJM, de Jonge WJM. 1995. Semiclassical calculations of the anisotropic magnetoresistance of NiFe-based thin films, wires, and multilayers. Phys. Rev. B 51, 283. (doi: 10.1103/PhysRevB.51.283)
  * (75) van Gorkom RP, Caro J, Klapwijk TM, Radelaar S. 2001. Temperature and angular dependence of the anisotropic magnetoresistance in epitaxial Fe films. Phys. Rev. B 63, 134432. (doi: 10.1103/PhysRevB.63.134432)
  * (76) Kimling J, Gooth J, Nielsch K. 2013. Anisotropic magnetothermal resistance in Ni nanowires. Phys. Rev. B 87, 094409. (doi: 10.1103/PhysRevB.87.094409)
  * (77) Xiao X, Li JX, Ding Z, Wu YZ. 2015. Four-fold symmetric anisotropic magnetoresistance of single-crystalline Ni(001) film. J. Appl. Phys. 118, 203905. (doi: 10.1063/1.4936175)
  * (78) I. Bakonyi et al. 2002. Europhys. Lett. 58, 408.
  * (79) Xiao X, Liang JH, Chen BL, Li JX, Ma DH, Ding Z, Wu YZ. 2015. Current-direction dependence of the transport properties in single-crystalline face-centered-cubic cobalt films. J. Appl. Phys. 118, 043908. (doi: 10.1063/1.4927620)
  * (80) Miao Y, Chen X, Yang S, Zheng K, Lian Z, Wang Y, Wang P, Gao C, Yang D-Z, Xue D-S. 2020. Non-cosine square angular-dependent magnetoresistance of the face-centered-cubic Co thin films. J. Magn. Magn. Mater. 512, 167013. (doi: 10.1016/j.jmmm.2020.167013)
  * (81) Levy P M, Fert A 2016. C. R. Physique 17, 447.
  * (82) McGuire T, Aboaf J, Klokholm E. 1984. Negative anisotropic magnetoresistance in 3d metals and alloys containing iridium. IEEE Trans Mag 20, 5. (doi: 10.1109/TMAG.1984.1063188)
  * (83) Ishio S, Haga H, Shindo S, Saito H. 1999. Anisotropic Magnetoresistance in Fe-Co-Ni Alloys. J. Magn. Soc. Japan 23, 427. doi: 10.3379/jmsjmag.23.427
  * (84) Berger L, Freitas PP, Warner JD, Schmidt JE. 1988. On the temperature dependence of the magnetoresistance of ferromagnetic alloys. J. Appl. Phys. 64, 5459. doi: 10.1063/1.342347
  * (85) Miyazaki T, Oikawa M. 1991. Magnetoresistance of Ni-Fe-Co ternary alloy films. J. Magn. Magn. Mater. 97, 171. (doi: 10.1016/0304-8853(91)90177-C)
  * (86) Wang S, Gao T, Wang C, He J. 2013. Studies of anisotropic magnetoresistance and magnetic property of Ni81Fe19 ultra-thin films with the lower base vacuum. J. All. Comp. 554, 405. (doi: 10.1016/j.jallcom.2012.12.004)
  * (87) Dahlberg ED, Riggs K. 1988. Magnetotransport: An ideal probe of anisotropy energies in epitaxial films (invited). J. Appl. Phys. 63, 4270. (doi: 10.1063/1.340200)
  * (88) Mašek J et al.. 2010. Microscopic Analysis of the Valence Band and Impurity Band Theories of (Ga,Mn)As. Phys. Rev. Lett. 105, 227202. (doi: 10.1103/PhysRevLett.105.227202)
  * (89) Baxter DV, Ruzmetov D, Scherschligt J, Sasaki Y, Liu X, Furdyna JK, Mielke CH. 2002. Anisotropic magnetoresistance in Ga1-xMnxAs. Phys. Rev. B 65, 212407. (doi: 10.1103/PhysRevB.65.212407)
  * (90) Wang KY, Edmonds KW, Campion RP, Zhao LX, Foxon CT, Gallagher BT. 2005. Anisotropic magnetoresistance and magnetic anisotropy in high-quality (Ga,Mn)As films. Phys. Rev. B 72, 085201. (doi: 10.1103/PhysRevB.72.085201)
  * (91) Miyakozawa S, Chen L, Matsukura F, Ohno H. 2016. Temperature dependence of in-plane magnetic anisotropy and anisotropic magnetoresistance in (Ga,Mn)As codoped with Li. Appl. Phys. Lett. 108, 112404. doi: 10.1063/1.4944328
  * (92) Howells B, Wang M, Edmonds KW, Wadley P, Campion RP, Rushforth AW, Foxon CT, Gallagher BL. 2013. Crystalline anisotropic magnetoresistance in quaternary ferromagnetic semiconductor (Ga,Mn)(As,Sb). Appl. Phys. Lett. 102, 052407. doi: 10.1063/1.4791580
  * (93) W. Wang et al. in RSC Adv., Effect of Sb content on anisotropic magnetoresistance in a (Ga, Mn)(As, Sb) ferromagnetic semiconductor thin film. RSC Adv. 9, 10776. doi: 10.1039/c8ra10256b
  * (94) Jungwirth T, Abolfath M, Sinova J, Ku??era J, MacDonald AH. 2002. Boltzmann theory of engineered anisotropic magnetoresistance in (Ga,Mn)As. Appl. Phys. Lett. 81, 4029. doi: 10.1063/1.1523160
  * (95) Jungwirth T et al. 2005. Prospects for high temperature ferromagnetism in (Ga,Mn)As semiconductors. Phys. Rev. B 72, 165204. (doi: 10.1103/PhysRevB.72.165204)
  * (96) Shapira Y, Oliveira, Jr. NF, Becla P, Vu TQ. 1990. Magnetoresistance and Hall effect near the metal-insulator transition of n-type Cd0.95Mn0.05Te. Phys. Rev. B 41, 5931. (doi: 10.1103/PhysRevB.41.5931)
  * (97) Navarro-Quezada A, Aiglinger M, Faina B, Gas K, Matzer M, Li T, Adhikari R, Sawicki M, Bonanni A. 2019. Magnetotransport in phase-separated (Ga,Fe)N with $\gamma^{\prime}-$GayFe4-yN nanocrystals. Phys. Rev. B 99, 085201. (doi: 10.1103/PhysRevB.99.085201)
  * (98) Lee JS, Richardella A, Rench DW, Fraleigh RD, Flanagan TC, Borchers JA, Tao J, Samarth N. 2014. Ferromagnetism and spin-dependent transport in n-type Mn-doped bismuth telluride thin films. Phys. Rev. B 89, 174425. (doi: 10.1103/PhysRevB.89.174425)
  * (99) Dyck JS, Drašar Č, Lošťák P, Uher C. 2005. Low-temperature ferromagnetic properties of the diluted magnetic semiconductor Sb2-xCrxTe3. Phys. Rev. B 71, 115214. (doi: 10.1103/PhysRevB.71.115214)
  * (100) Lee JW, Kuroda S, Takano F, Akinaga H, Takita K. 2006. Anisotropy of magnetization and magnetoresistance of (Zn,Co)O films grown by pulsed laser deposition. phys. stat. sol. (c) 3, 4098. (doi: 10.1002/pssc.200672872)
  * (101) Khalid M, Esquinazi P. 2012. Hydrogen-induced ferromagnetism in ZnO single crystals investigated by magnetotransport. Phys. Rev. B 85, 134424. (doi: 10.1103/PhysRevB.85.134424)
  * (102) Landau LD. 1933. A possible explanation of the field dependence of the susceptibility at low temperatures. Phys. Z. Sowjet 4, 675. (doi: 10.1016/B978-0-08-010586-4.50017-1)
  * (103) Fina I et al. 2014. Anisotropic magnetoresistance in an antiferromagnetic semiconductor. Nat. Commun. 5, 4671. (doi: 10.1038/ncomms5671)
  * (104) Wang C, Seinige H, Cao G, Zhou J-S, Goodenough JB, Tsoi M. 2014. Anisotropic Magnetoresistance in Antiferromagnetic Sr2IrO4. Phys. Rev. X 4, 041034. doi: 10.1103/PhysRevX.4.041034
  * (105) X. Marti et al. 2014. Room-temperature antiferromagnetic memory resistor. Nature Mater. 13, 367. (doi: 10.1038/nmat3861)
  * (106) Kudrnovský J, Drchal V, Turek I. 2015. Physical properties of FeRh alloys: The antiferromagnetic to ferromagnetic transition. Phys. Rev. B 91, 014435. (doi: 10.1103/PhysRevB.91.014435)
  * (107) Kabara K, Tsunoda M, Kokado S. 2014. Annealing effects on nitrogen site ordering and anisotropic magnetoresistance in pseudo-single-crystal $\gamma^{\prime}$-Fe4N films. Appl. Phys. Express 7, 063003. (doi: 10.7567/APEX.7.063003)
  * (108) Baltz V, Manchon A, Tsoi M, Moriyama T, Ono T, Tserkovnyak Y. 2018. Antiferromagnetic spintronics. Rev. Mod. Phys. 90, 015005. (doi: 10.1103/RevModPhys.90.015005)
  * (109) Wadley P et al. 2016. Electrical switching of an antiferromagnet. Science 351, 6273. (doi: 10.1126/science.aab1031)
  * (110) Zubáč J, Kašpar Z, Krizek F, Förster T, Campion RP, Nov??k V, Jungwirth T, Olejník K. Hysteretic effects and magnetotransport of electrically switched CuMnAs. Phys. Rev. B 104, 184424. (doi: 10.1103/PhysRevB.104.184424)
  * (111) Emmanouilidou E, Cao H, Tang P, Gui X, Hu C, Shen B, Wu J, Zhang S-C, Xie W, Ni N. 2017. Magnetic order induces symmetry breaking in the single-crystalline orthorhombic CuMnAs semimetal. Phys. Rev. B 96, 224405. (doi: 10.1103/PhysRevB.96.224405)
  * (112) Bodnar SY, Šmejkal L, Turek I, Jungwirth T, Gomonay O, Sinova J, Sapozhnik AA, Elmers H-J, Kläui M, Jourdan M. 2018. Writing and reading antiferromagnetic Mn2Au by N??el spin-orbit torques and large anisotropic magnetoresistance. Nat. Commun. 9, 348. doi: 10.1038/s41467-017-02780-x
  * (113) Manna K, Sun Y, Muechler L, Kübler J, Felser C. 2018. Heusler, Weyl and Berry. Nature Reviews Materials 3, 244-256. (doi: 10.1038/s41578-018-0036-5)
  * (114) Heusler F. 1903. Über magnetische Manganlegierungen. Verhandlungen der Deutschen Physikalischen Gesellschaft 5, 219.  (doi?)
  * (115) Sakuraba Y, Kokado S, Hirayama Y, Furubayashi T, Sukegawa H, Li S, Takahashi YK, Hono K. 2014. Quantitative analysis of anisotropic magnetoresistance in Co2MnZ and Co2FeZ epitaxial thin films: A facile way to investigate spin-polarization in half-metallic Heusler compounds. Appl. Phys. Lett. 104, 172407. (doi: 10.1063/1.4874851)
  * (116) Yako H, Kubota T, Takanashi K. 2015. Anisotropic Magnetoresistance Effect in Co2(Fe,Mn)(Al,Si) Heusler Alloy Thin Film. IEEE Trans Mag, 51, 11. (doi: 10.1109/TMAG.2015.2439284)
  * (117) Oogane M, McFadden AP, Kota Y, Brown-Heft TL, Tsunoda M, Ando Y, Palmström CJ. 2018. Fourfold symmetric anisotropic magnetoresistance in half-metallic Co2MnSi Heusler alloy thin films. Jpn. J. Appl. Phys. 57, 063001. (doi: 10.7567/JJAP.57.063001)
  * (118) Felser C, Hirohata A. 2016. Heusler alloys: Properties, Growth, Applications. Springer. (ISBN: 978-3-319-21449-8.)
  * (119) Ciccarelli C et al. 2016. Room-temperature spin-orbit torque in NiMnSb. Nature Phys. 12, 855. doi: 10.1038/nphys3772
  * (120) Ohtomo A, Hwang HY. 2004. A high-mobility electron gas at the LaAlO3/SrTiO3 heterointerface. Nature 427, 423. (doi: 10.1038/nature02308)
  * (121) Shalom MB, Tai CW, Lereah Y, Sachs M, Levy E, Rakhmilevitch D, Palevski A, Dagan Y. Anisotropic magnetotransport at the SrTiO3/LaAlO3 interface. Phys. Rev. B 80, 140403(R). (doi: 10.1103/PhysRevB.80.140403)
  * (122) Reyren N et al. 2007\. Superconducting Interfaces Between Insulating Oxides. Science 317, 1196. (doi: 10.1126/science.1146006)
  * (123) Huijben M, Brinkman A, Koster G, Rijnders G, Hilgenkamp H, Blank DHA. 2009. Structure–Property Relation of SrTiO3/LaAlO3 Interfaces. Adv. Mater. 21, 1665. (doi: 10.1002/adma.200801448)
  * (124) Lebedev N, Stehno M, Rana A, Gauquelin N, Verbeeck J, Brinkman A, Aarts J. 2020. Inhomogeneous superconductivity and quasilinear magnetoresistance at amorphous LaTiO3/SrTiO3 interfaces. J. Phys.: Condens. Matter 33, 055001. (doi: 10.1088/1361-648X/abc102)
  * (125) Boschker H, Mannhart J. 2017. Quantum-Matter Heterostructures. Annu. Rev. Condens. Matter Phys. 8:145???64. (doi: 10.1146/annurev-conmatphys-031016-025404)
  * (126) Bovenzi N, Diez M. 2017. Semiclassical theory of anisotropic transport at LaAlO3/SrTiO3 interfaces under an in-plane magnetic field. Phys. Rev. B 95, 205430. (doi: 10.1103/PhysRevB.95.205430)
  * (127) Ariando, Wang X et al. 2011. Electronic phase separation at the LaAlO3/SrTiO3 interface. Nat. Commun. 2, 188. (doi: 10.1038/ncomms1192)
  * (128) Rout PK, Agireen I, Maniv E, Goldstein M, Dagan Y. 2017. Six-fold crystalline anisotropic magnetoresistance in the (111) LaAlO3/SrTiO3 oxide interface. Phys. Rev. B 95, 241107(R). (doi: 10.1103/PhysRevB.95.241107)
  * (129) Joshua A, Ruhman J, Pecker S, Altman E, Ilani S. 2013. Gate-tunable polarized phase of two-dimensional electrons at the LaAlO3/SrTiO3 interface. Proc. Natl. Acad. Sci. USA 110, 9633. (doi: 10.1073/pnas.1221453110)
  * (130) Miao L, Du R, Yin Y, Li Q. 2016. Anisotropic magneto-transport properties of electron gases at SrTiO3 (111) and (110) surfaces. Appl. Phys. Lett. 109, 261604. (doi: 10.1063/1.4972985)
  * (131) Harsan Ma HJ, Zhou J, Yang M, Liu Y, Zeng SW, Zhou WX, Zhang LC, Venkatesan T, Feng YP, Ariando. 2017. Giant crystalline anisotropic magnetoresistance in nonmagnetic perovskite oxide heterostructures. Phys. Rev. B 95, 155314. (doi: 10.1103/PhysRevB.95.155314)
  * (132) Boudjada N, Khait I, Paramekanti A. 2019. Anisotropic magnetoresistance in multiband systems: Two-dimensional electron gases and polar metals at oxide interfaces. Phys. Rev. B 99, 195453. (doi: 10.1103/PhysRevB.99.195453)
  * (133) Wadehra N, Tomar R, Varma RM, Gopal RK, Singh Y, Dattagupta S, Chakraverty S. 2020. Planar Hall effect and anisotropic magnetoresistance in polar-polar interface of LaVO3-KTaO3 with strong spin-orbit coupling. Nat. Commun. 11, 874. (doi: 10.1038/s41467-020-14689-z)
  * (134) Tomar R, Kakkar S, Bera C, Chakraverty S. 2021. Anisotropic magnetoresistance and planar Hall effect in (001) and (111) LaVO3 / SrTiO3 heterostructures. Phys. Rev. B 103, 115407. (doi: 10.1103/PhysRevB.103.115407)
  * (135) Chen Y et al. 2015. Creation of High Mobility Two-Dimensional Electron Gases via Strain Induced Polarization at an Otherwise Nonpolar Complex Oxide Interface. Nano Lett. 15, 1849. (doi: 10.1021/nl504622w)
  * (136) Granada M, Bustingorry S, Pontello DE, Barturen M, Eddrief M, Marangolo M, Milano J. 2016. Magnetotransport properties of Fe0.8Ga0.2 films with stripe domains. Phys. Rev. B 94, 184435. (doi: 10.1103/PhysRevB.94.184435)
  * (137) Jen SU. 1992. Anisotropic magnetoresistance of Co-Pd alloys. Phys. Rev. B 45, 9819. (doi: 10.1103/PhysRevB.45.9819)
  * (138) Philippi-Kobs A, Farhadi A, Matheis L, Lott D, Chuvilin A, Oepen HP. 2019. Impact of Symmetry on Anisotropic Magnetoresistance in Textured Ferromagnetic Thin Films. Phys. Rev. Lett. 123, 137201. (doi: 10.1103/PhysRevLett.123.137201)
  * (139) T. Hupfauer, A. Matos-Abiague, M. Gmitra, F. Schiller, J. Loher, D. Bougeard, C. H. Back, J. Fabian, and D. Weiss, Emergence of spin-orbit fields in magnetotransport of quasi-two-dimensional iron on gallium arsenide, Nat. Commun. 6, 7374 (2015).
  * (140) Shi X, Li X, Lai Z, Liu X, Mi W. 2020. Structure, magnetic and electronic transport properties in antiperovskite cubic $\gamma^{\prime}$-CuFe3N polycrystalline films. Intermetallics 121, 106779. (doi: 10.1016/j.intermet.2020.106779)
  * (141) Tsunoda M, Komasaki Y, Kokado S, Isogami S, Chen C-C, Takahashi M. 2009. Negative Anisotropic Magnetoresistance in Fe4N Film. Appl. Phys. Express 2, 083001. (doi: 10.1143/APEX.2.083001)
  * (142) Tsunoda M, Takahashi H, Kokado S, Komasaki Y, Sakuma A, Takahashi M. 2010. Anomalous Anisotropic Magnetoresistance in Pseudo-Single-Crystal $\gamma^{\prime}$-Fe4N Films. Appl. Phys. Express 3, 113003. (doi: 10.1143/APEX.3.113003)
  * (143) Klein L, Marshall AF, Reiner JW, Ahn CH, Geballe TH, Beasley MR, Kapitulnik A. 1998. Large magnetoresistance of single-crystal films of ferromagnetic SrRuO3. J. Magn. Magn. Mater. 188, 319. (doi: 10.1016/S0304-8853(98)00201-7)
  * (144) Herranz G, Sanchez F, Garcia-Cuenca MV, Ferrater C, Varela M, Martinez B; Fontcuberta J. 2004. Anisotropic magnetoresistance in SrRuO3 ferromagnetic oxide. J. Magn. Magn. Mater. 272-276, 517. (doi: 10.1016/j.jmmm.2003.12.1050)
  * (145) Rao RA, Kacedon DB, Eom CB. 1998. Anisotropic magnetotransport properties of epitaxial thin films of conductive ferromagnetic oxide SrRuO3. J. Appl. Phys. 83, 6995. (doi: 10.1063/1.367674)
  * (146) Haham N, Shperber Y, Reiner JW, Klein L. 2013. Low-temperatrure anisotropic magnetoresistance and planar Hall effect in SrRuO3. Phys. Rev. B. 87, 144407. (doi: 10.1103/PhysRevB.87.144407)
  * (147) Chaurasia R, Asokan K, Kumar K, Pramanik AK. 2021. Low-temperature ferromagnetism in perovskite SrIrO3 films. Phys. Rev. B 103, 064418. (doi: 10.1103/PhysRevB.103.064418)
  * (148) Zeng Z, Greenblatt M, and Croft M. 1999. Large magnetoresistance in antiferromagnetic CaMnO3-δ. Phys. Rev. B 59, 8784. (doi: 10.1103/PhysRevB.59.8784)
  * (149) Li X et al. 2021\. Charge Disproportionation and Complex Magnetism in a PbMnO3 Perovskite Synthesized under High Pressure. Chem. Mater. 33, 1, 92???101. (doi: 10.1021/acs.chemmater.0c02706)
  * (150) Bibes M, Laukhin V, Valencia S, Martinez B, Fontcuberta J, Gorbenko OY, Kaul AR, Martinez JL. 2005. Anisotropic magnetoresistance and anomalous Hall effect in manganite thin films. J. Phys.: Condens. Matter 17 2733\. (doi: http://dx.doi.org/10.1088/0953-8984/17/17/022)
  * (151) O’Donnell J, Eckstein JN, Rzchowski MS. 2000. Temperature and magnetic field dependent transport anisotropies in La0.7Ca0.3MnO3 films. Appl. Phys. Lett. 76, 218. (https://doi.org/10.1063/1.125707)
  * (152) Yang S, Chen Q, Yang Y, Gao Y, Xu R, Zhang H, Ma J. 2021. Silver addition in polycrystalline La0.7Ca0.3MnO3: Large magnetoresistance and anisotropic magnetoresistance for manganite sensors. J. Alloys Compd. 882, 160719. (doi: 10.1016/j.jallcom.2021.160719)
  * (153) Sharma H, Tulapurkar A, Tomy CV. 2014. Sign reversal of anisotropic magnetoresistance in La0.7Ca0.3MnO3/SrTiO3 ultrathin films. Appl. Phys. Lett. 105, 222406. (doi: 10.1063/1.4903236)
  * (154) Xie Y, Yang H, Liu Y, Yang Z, Chen B, Zuo Z, Katlakunta S, Zhan Q, Li R-W. 2013. Strain induced tunable anisotropic magnetoresistance in La0.67Ca0.33MnO3/BaTiO3 heterostructures. J Appl Phys 113, 17C716. (doi: 10.1063/1.4795841)
  * (155) Kandpal LM, Singh S, Kumar P, Siwach PK, Gupta A, Awana VPS, Singh HK. 2016. Magnetic anisotropy and anisotropic magnetoresistance in strongly phase separated manganite thin films. J. Magn. Magn. Mater. 408, 60. (doi: 10.1016/j.jmmm.2016.02.022)
  * (156) Egilmez M, Saber MM, Mansour AI, Ma R, Chow KH, Jung J. 2008. Dramatic strain induced modification of the low field anisotropic magnetoresistance in ultrathin manganite films. Appl. Phys. Lett. 93, 182505. (doi: 10.1063/1.3021083)
  * (157) Wong AT, Beekman C, Guo H, Siemons W, Gai Z, Arenholz E, Takamura Y, Ward TZ. 2014. Strain driven anisotropic magnetoresistance in antiferromagnetic La0.4Sr0.6MnO3. Appl. Phys. Lett. 105, 052401. (doi: 10.1063/1.4892420)
  * (158) Infante IC, Laukhin V, Sanchez F, Fotcuberta J, Melnikov O, Gorbenko OY, Kaul AR. 2006. Anisotropic magnetoresistance in epitaxial (110) manganite films. J. Appl. Phys. 99, 08C502. (https://doi.org/10.1063/1.2150812)
  * (159) Kumar P, Prasad R, Dwivedi RK, Singh HK. 2011. Out-of-plane low field anisotropic magnetoresistance in Nd0.51Sr0.49MnO3 thin films. J. Magn. Magn. Mater. 323, 2564. (doi: 10.1016/j.jmmm.2011.05.032)
  * (160) Chen YZ, Sun JR, Zhao TY, Wang J, Wang ZH, Shen BG, Pryds N. 2009. Crossover of angular dependent magnetoresistance with the metal-insulator transition in colossal magnetoresistive manganite films. Appl. Phys. Lett. 95, 132506. (doi: 10.1063/1.3240407)
  * (161) Li P, Jin C, Jiang EY, Bai HL. 2010. Origin of the twofold and fourfold symmetric anisotropic magnetoresistance in epitaxial. J. Appl. Phys. 108, 093921. (doi: 10.1063/1.3499696)
  * (162) Bai H et al. 2022. Observation of Spin Splitting Torque in a Collinear Antiferromagnet RuO2. Phys. Rev. Lett. 128, 197202. (doi: 10.1103/PhysRevLett.128.197202)
  * (163) Fang D, Kurebayashi H, Wunderlich J, Vyborny K, Zarbo LP, Campion RP, Casiraghi A, Gallagher BL, Jungwirth T, Ferguson AJ. 2011. Spin–orbit-driven ferromagnetic resonance. Nature Nanotech. 6, 413. (doi: 10.1038/nnano.2011.68)
  * (164) Mellnik AR, Lee JS, Richardella A, Grab JL, Mintun PJ, Fischer MH, Vaezi A, Manchon A, Kim E-A, Samarth N, Ralph DC. 2014. Spin-transfer torque generated by a topological insulator. Nature 511, 449. (doi: 10.1038/nature13534)
  * (165) Booth K, Gray I, Dahlberg ED. 2021. Determining the AC susceptibility of thin metal films using the anisotropic magnetoresistance. J. Magn. Magn. Mater. 523, 167631. (doi: 10.1016/j.jmmm.2020.167631)
  * (166) Pignard S, Goglio G, Radulescu A, Piraux L, Dubois S, Declemy A, Duvail JL. 2002. Study of the magnetization reversal in individual nickel nanowires. J. Appl. Phys. 87, 824. (doi: 10.1063/1.371947)
  * (167) Wegrowe J-E, Kelly D, Franck A, Gilbert SE, Ansermet J-Ph. 1999. Magnetoresistance of Ferromagnetic Nanowires. Phys. Rev. Lett. 82, 3681. (doi: 10.1103/PhysRevLett.82.3681.)
  * (168) Rheem Y, Yoo B-Y, Beyermann WP, Myung NV. 2006. Magnetotransport studies of a single nickel nanowire. Nanotechnology 18, 015202. (doi: 10.1088/0957-4484/18/1/015202)
  * (169) Hayashi M, Thomas L, Rettner C, Moriya R, Jiang X, Parkin SP. 2006. Dependence of Current and Field Driven Depinning of Domain Walls on Their Structure and Chirality in Permalloy Nanowires. Phys. Rev. Lett 97, 207205. (doi: 10.1103/PhysRevLett.97.207205)
  * (170) Nam Hai P, Duc Anh L, Tanaka M. 2012. Electron effective mass in n-type electron-induced ferromagnetic semiconductor (In,Fe)As: Evidence of conduction band transport. Appl. Phys. Lett. 101, 252410. (doi: 10.1063/1.4772630)
  * (171) Campbell IA, Fert A, Jaoul O. 1970. The spontaneous resistivity anisotropy in Ni-based alloys. J. Phys. C 3, S95. (doi: 10.1088/0022-3719/3/1S/310)
  * (172) Berger L. 2011. Spin relaxation in metallic ferromagnets. Phys. Rev. B 83, 054410. (doi: 10.1103/PhysRevB.83.054410)
  * (173) Narayanapillai K, Gopinadhan K, Qiu X, Annadi A, Ariando, Venkatesan T, Yang H. 2014. Current-driven spin orbit field in LaAlO3/SrTiO3 heterostructures. Appl. Phys. Lett. 105, 162405. (doi: 10.1063/1.4899122)
  * (174) Feng Z et al. 2012. Spin Hall angle quantification from spin pumping and microwave photoresistance. Phys. Rev. B 85, 214423. (doi: 10.1103/PhysRevB.85.214423)
  * (175) Mosendz O, Pearson JE, Fradin FY, Bauer GEW, Bader SD, Hoffmann A. 2010. Quantifying Spin Hall Angles from Spin Pumping: Experiments and Theory. Phys. Rev. Lett. 104, 046601. (doi: 10.1103/PhysRevLett.104.046601)
  * (176) Xing X, Xu C, Li Z, Feng J, Zhou N, Zhang Y, Sun Y, Zhou W, Xu X, Sh Z. 2018. Angular-dependent magnetoresistance study in Ca0.73La0.27FeAs2: a ’parent’ compound of 112-type iron pnictide superconductors. J. Phys.: Condens. Matter 30, 025701. (doi: 10.1088/1361-648X/aa9c11)
  * (177) Park J-H, Ko H-W, Kim J-M, Park J, Park S-Y, Jo Y, Park B-G, Kim SK, Lee K-J, Kim K-J. 2021. Temperature dependence of intrinsic and extrinsic contributions to anisotropic magnetoresistance. Sci. Rep. 11, 20884. (doi: 10.1038/s41598-021-00374-8)
  * (178) Tesařová N et al. 2014. Systematic study of magnetic linear dichroism and birefringence in (Ga,Mn)As. Phys. Rev. B 89, 085203. (doi: 10.1103/PhysRevB.89.085203)
  * (179) Silber R et al. 2019. Quadratic magneto-optic Kerr effect spectroscopy of Fe epitaxial films on MgO(001) substrates. Phys. Rev. B 100, 064403. (doi: 10.1103/PhysRevB.100.064403)
  * (180) Hamrle J, Blomeier S, Gaier O, Hillebrands B, Schneider H, Jakob G, Postava K, Felser C. 2007. Huge quadratic magneto-optical Kerr effect and magnetization reversal in the Co2FeSi Heusler compound. J. Phys. D: Appl. Phys. 40, 1563. doi: 10.1088/0022-3727/40/6/S09
  * (181) Valencia S, Kleibert A, Gaupp A, Rusz J, Legut D, Bansmann J, Gudat W, Oppeneer PM. 2010. Quadratic X-Ray Magneto-Optical Effect upon Reflection in a Near-Normal-Incidence Configuration at the M Edges of 3d-Transition Metals. Phys. Rev. Lett. 104, 187401. doi: 10.1103/PhysRevLett.104.187401
  * (182) Zink BL. 2022. Thermal effects in spintronic materials and devices: An experimentalist???s guide. J. Magn. Magn. Mater. 564, 170120. (doi: 10.1016/j.jmmm.2022.170120)
  * (183) Slachter A, Bakker FL, van Wees BJ. 2011. Modeling of thermal spin transport and spin-orbit effects in ferromagnetic/nonmagnetic mesoscopic devices. Phys. Rev. B 84, 174408. (doi: 10.1103/PhysRevB.84.174408)
  * (184) Wegrowe J-E, Drouhin H-J, Lacour D. 2014. Anisotropic magnetothermal transport and spin Seebeck effect. Phys. Rev. B 89, 094409. (doi: 10.1103/PhysRevB.89.094409)
  * (185) Jungwirth T et al. 2014\. Spin-dependent phenomena and device concepts explored in (Ga,Mn)As. Rev. Mod. Phys. 86, 855. (doi: 10.1103/RevModPhys.86.855)
  * (186) Kimling J, Gooth J, Nielsch K. 2013. Anisotropic magnetothermal resistance in Ni nanowires. Phys. Rev. B 87, 094409. (doi: 10.1103/PhysRevB.87.094409)
  * (187) Heikkilä TT, Hatami M, Bauer GEW. 2010. Spin heat accumulation and its relaxation in spin valves. Phys. Rev. B 81, 100408(R). (doi: 10.1103/PhysRevB.81.100408)
  * (188) Corliss LM, Elliott N, Hastings JM, Sass RL. 1961. Magnetic Structure of Chromium Selenide. Phys. Rev. 122, 1402. (doi: 10.1103/PhysRev.122.1402)
  * (189) Shindou R, Nagaosa N. 2001. Orbital Ferromagnetism and Anomalous Hall Effect in Antiferromagnets on the Distorted fcc Lattice. Phys. Rev. Lett. 87, 116801. (doi: 10.1103/PhysRevLett.87.116801)
  * (190) Chen H, Niu Q, MacDonald AH. 2014. Anomalous Hall Effect Arising from Noncollinear Antiferromagnetism. Phys. Rev. Lett. 112, 017205. (doi: 10.1103/PhysRevLett.112.017205)
  * (191) Velev J, Sabirianov RF, Jaswal SS, Tsymbal EY. 2005. Ballistic Anisotropic Magnetoresistance. Phys. Rev. Lett. 94, 127203. (doi: 10.1103/PhysRevLett.94.127203)
  * (192) Hu C, Teng J, Yu G, Lu W, Ji W. 2015. Conditions for quantized anisotropic magnetoresistance. Phys. Rev. B 91, 045438. (doi: 10.1103/PhysRevB.91.045438)
  * (193) Autes G, Barreteau C, Spanjaard D, Desjonqueres M-C. 2008. Electronic transport in iron atomic contacts: From the infinite wire to realistic geometries. Phys. Rev. B 77, 155437. (doi: 10.1103/PhysRevB.77.155437)
  * (194) Moodera JS, Kinder LR, Wong TM, Meservey R. 1995. Large Magnetoresistance at Room Temperature in Ferromagnetic Thin Film Tunnel Junctions. Phys. Rev. Lett. 74, 3273. (doi: 10.1103/PhysRevLett.74.3273)
  * (195) Julliere M. 1975. Tunneling between ferromagnetic films. Phys. Lett. A 54, 225. (doi: 10.1016/0375-9601(75)90174-7)
  * (196) Gould C et al. 2004\. Tunneling Anisotropic Magnetoresistance: A Spin-Valve-Like Tunnel Magnetoresistance Using a Single Magnetic Layer. Phys. Rev. Lett. 93, 117203. (doi: 10.1103/PhysRevLett.93.117203)
  * (197) Schöneberg J, Ferriani P, Heinze S, Weismann A, Berndt R. 2018. Tunneling anisotropic magnetoresistance via molecular $\pi$ orbitals of Pb dimers. Phys. Rev. B 97, 041114. (doi: 10.1103/PhysRevB.97.041114)
  * (198) Kandala A, Richardella A, Kempinger S, Liu C-X, Samarth N. 2015. Giant anisotropic magnetoresistance in a quantum anomalous Hall insulator. Nat. Commun. 6, 7434, (doi: 10.1038/ncomms8434)
  * (199) Heidecker J. 2013. MRAM Technology Status. JPL Publication 13-3. [cited 2022 Nov 11] Available from: https://ntrs.nasa.gov/api/citations/20140000668/downloads/20140000668.pdf
  * (200) Bartok A, Daniel L, Razek A. 2013. A multiscale model for thin film AMR sensors. J. Magn. Magn. Mater. 326, 116. (doi: 10.1016/j.jmmm.2012.08.020)
  * (201) Stutzke NA, Russek SE, Pappas DP, Tondra M. 2005. Low-frequency noise measurements on commercial magnetoresistive magnetic field sensors. J. Appl. Phys. 97, 10Q107. (doi: 10.1063/1.1861375)
  * (202) Honeywell. Magnetic Displacement Sensors. Technical Report HMC 1501-1512. [cited 2022 Nov 11] Available from: https://aerospace.honeywell.com/content/dam/aerobt/en/documents/learn/products/sensors/datasheet/N61-2042-000-000_MDS_HMC15011512-ds.pdf
  * (203) Philips Semiconductors. 2000. Application Note, General Magnetoresistive Sensors for Magnetic Field Measurement, Technical Report KMZ10. [cited 2022 Nov 11] Available from: https://www.mikrocontroller.net/attachment/27041/SC17_GENERAL_MAG_2-1.pdf
  * (204) Adelerhof DJ, Geven W. 2000. New position detectors based on AMR sensors. Sens. Actuator A Phys. 85, 48. (doi: 10.1016/S0924-4247(00)00341-1)
  * (205) Včelák J, Ripka P, Kubík J, Platil A, Kašpar P. 2005. AMR navigation systems and methods of their calibration. Sens. Actuator A Phys. 123-124, 122. (doi: 10.1016/j.sna.2005.02.040)
  * (206) Mlejnek P, Vopalensky M, Ripka P. 2008. AMR current measurement device. Sens. Actuator A Phys. 141, 649. (doi: 10.1016/j.sna.2007.10.016)
  * (207) Schuhl A, Van Dau FN, Childress JR. 1995. Low-field magnetic sensors based on the planar Hall effect. Appl. Phys. Lett. 66, 2751. (doi: 10.1063/1.113697)
  * (208) Jungwirth T, Niu Q, MacDonald AH. 2002. Anomalous Hall Effect in Ferromagnetic Semiconductors. Phys. Rev. Lett. 88, 207208. (doi: 10.1103/PhysRevLett.88.207208)
  * (209) Breidenbach AT, Yu H, Peterson TA, McFadden AP, Peria WK, Palmström CJ, Crowell PA. 2022. Anomalous Nernst and Seebeck coefficients in epitaxial thin film Co2MnAlxSi1-x and Co2FeAl. Phys. Rev. B 105, 144405. (doi: 10.1103/PhysRevB.105.144405)


# Distributed Dual Coordinate Ascent with Imbalanced Data on a General Tree
Network

###### Abstract

In this paper, we investigate the impact of imbalanced data on the convergence
of distributed dual coordinate ascent in a tree network for solving an
empirical loss minimization problem in distributed machine learning. To
address this issue, we propose a method called delayed generalized distributed
dual coordinate ascent that takes into account the information of the
imbalanced data, and provide the analysis of the proposed algorithm. Numerical
experiments confirm the effectiveness of our proposed method in improving the
convergence speed of distributed dual coordinate ascent in a tree network.

Index Terms—  distributed machine learning, federated learning, tree network,
network topology, imbalanced data

##  1 Introduction

In the field of Machine Learning (ML) and Artificial Intelligence (AI), the
use of large amounts of data, known as big data, plays a crucial role in the
performance of ML and AI techniques. Due to the exceptional performance of
ML/AI with big data, these techniques are becoming increasingly popular and
are being applied to many different applications.

However, in practice, processing ML/AI operations with big data poses many
challenges such as limited hardware resources including storage space and
processing power, and privacy and security issues. Specifically, due to
limited storage space, big data is often stored in a distributed manner over a
network, raising questions about how to deal with these distributed data when
processing ML/AI operations. Furthermore, even though there are distributed
algorithms that can process distributed data for ML/AI operations,
communication to share intermediate results such as learning parameters can be
a significant challenge due to constraints such as limited communication
bandwidth, delay, and power. Thus, it is important to design efficient
algorithms for distributed ML/AI processes that take into account these
communication network constraints when dealing with distributed data.

To address the challenge of handling distributed data on a network, many
research were conducted to develop efficient distributed ML/AI algorithms.
Researchers in [1, 2, 3, 4, 5] studied synchronous Stochastic Gradient Decent
(SGD). Synchronous Stochastic Dual Coordinate Ascent (SDCA) and its variations
were investigated in [6, 7, 8, 9, 10, 11, 12, 13]. Asynchronous SGD was
studied in [14, 15, 16]. Asynchronous SDCA was investigated for handling
distributed data on a network in [17, 18, 19, 20].

However, most of the research in this area has focused on designing
distributed algorithms on a star network, which is a very simple network
topology. It’s worth noting that data are not always stored on a star network.
In reality, networks can have various topologies such as star, tree, ring,
bus, or mesh, etc. Additionally, in a network, some nodes may not directly
communicate with a central node. In this case, a line of nodes may be
considered as a virtual node directly connected to a central node to apply
distributed algorithms for a star network. However, the system can easily
suffer from significant communication delays caused by passing intermediate
results through the line of nodes to a central node. Thus, it is important to
design efficient distributed ML/AI algorithms by considering different network
topologies. To address this problem, the researchers in [12, 13] have designed
a distributed dual coordinate ascent for distributed ML process on a general
tree network, and provided the convergence analysis of the algorithm on a
general tree network under the assumption that the dataset is evenly
distributed on a tree network. Since every connected network has its spanning
tree, the distributed algorithm can be applied to various network topologies
as long as there are no isolated nodes in the network.

In practice, however, due to different circumstances and situations in data
acquisition such as different frequency of data acquisition in nodes,
different sensitivity of sensors, and noise or bias during data acquisition,
we can have imbalanced data on a network in distributed ML process. In this
paper, we investigate the effect of imbalanced data on distributed dual
coordinate ascent in a general tree network, and propose our method called a
generalized distributed dual coordinate ascent on a general tree network to
mitigate the effect of imbalanced data on processing distributed data on a
tree network.

Notations: We denoted the set of real numbers as ${\mathbb{R}}$. We use $[k]$
to denote the index set of the coordinates in the $k$-th coordinate block. For
an index set $Q$, $|Q|$ and $\overline{Q}$ represent the cardinality of $Q$
and the complement of $Q$ respectively. We reserve bold letters to represent
vectors and matrices. If an index set is used as a subscript of a vector
(resp. matrix), it indicates the partial vector (resp. partial matrix) over
the index set (resp. with columns over the index set). We use the superscript
$(t)$ to denote the $t$-th iteration. For instance,
${\bm{\alpha}}^{(t)}_{[k]}$ represents a partial vector ${\bm{\alpha}}$ over
the $k$-th block coordinate set at the $t$-th iteration. The superscript
$\star$ is reserved to denote the optimal solution to an optimization problem.

##  2 Problem Formulation

We perform an ML operation on a distributed dataset, denoted as
$\\{({\bm{x}}_{i},y_{i})\\}_{i=1}^{m}$, where
${\bm{x}}_{i}\in{\mathbb{R}}^{d}$ represents the $i$-th data point and $y_{i}$
is the measurement or label information associated with it. Our dataset is
distributed in a tree-shaped network consisting of $K$ local workers, as
illustrated in Fig. 1. The $k$-th local worker holds a subset of the dataset,
specifically $\\{({\bm{x}}_{i},y_{i})\\}_{i\in[k]}$ where $|[k]|<m$ and
$k=1,2,...,K$. Our objective is to find the global optimal solution,
${\bm{w}}^{\star}$, by solving the following regularized loss minimization
problem with the distributed dataset:

$\displaystyle\underset{{\bm{w}}\in{\mathbb{R}}^{d}}{\text{minimize}}\;P({\bm{w}})\triangleq\frac{\lambda}{2}\|{\bm{w}}\|^{2}_{2}+\frac{1}{m}\sum_{i=1}^{m}\ell_{i}({\bm{w}}^{T}{\bm{x}}_{i}),$
(1)

where $\ell_{i}(\cdot)$, $i=1,2,...,m$, are loss functions, and $\lambda\geq
0$ is a tuning parameter. Depending on the loss functions, the optimization
problem (1) can be a regression problem or a classification problem. For
instance, when the loss function is
$\ell_{i}({\bm{w}}^{T}{\bm{x}}_{i})=({\bm{w}}^{T}{\bm{x}}_{i}-y_{i})^{2}$ with
measurement data $y_{i}\in{\mathbb{R}}$, the problem can be interpreted as a
regression problem. When the loss function is
$\ell_{i}({\bm{w}}^{T}{\bm{x}}_{i})=\max(0,1-y_{i}({\bm{w}}^{T}{\bm{x}}_{i}))$
with label information $y_{i}\in{\mathbb{R}}$, (1) becomes a Support Vector
Machine (SVM) classification problem. Furthermore, we assume that the data
points are normalized to ensure that the $\ell_{2}$ norm of each
${\bm{x}}_{i}$ is bounded, i.e., $\|{\bm{x}}_{i}\|_{2}\leq 1$, for
$i=1,2,...,m$.

By considering the conjugate function of the loss function $\ell_{i}(a)$,
defined as $\ell_{i}(a)=\sup_{b}ab-\ell_{i}^{*}(b)$, we can derive the
following dual problem from the primal problem (1):

$\displaystyle\underset{{\bm{\alpha}}\in{\mathbb{R}}^{m}}{\text{maximize}}\;D({\bm{\alpha}})\triangleq-\frac{\lambda}{2}\|{\bm{A}}{\bm{\alpha}}\|^{2}_{2}-\frac{1}{m}\sum_{i=1}^{m}\ell^{*}_{i}(-\alpha_{i}),$
(2)

where ${\bm{A}}\in{\mathbb{R}}^{d\times m}$ is a data matrix whose $i$-th
column is $\frac{1}{\lambda m}{\bm{x}}_{i}$, and $\alpha_{i}$ is the $i$-th
dual variable corresponding to the $i$-th datum ${\bm{x}}_{i}$, $i=1,...,m$.
By defining ${\bm{w}}({\bm{\alpha}})\triangleq{\bm{A}}{\bm{\alpha}}$, we can
have a duality gap as $P({\bm{w}}({\bm{\alpha}}))-D({\bm{\alpha}})$ which can
be used as a measurable quantity for how close an estimated solution is to an
optimal solution. Remark that the weak duality theorem [21] ensures that for
all ${\bm{w}}$, $P({\bm{w}})\geq D({\bm{\alpha}})$. When we substitute
${\bm{w}}$ with ${\bm{w}}({\bm{\alpha}})$, this condition still holds with
$P({\bm{w}}({\bm{\alpha}}))\geq D({\bm{\alpha}})$. If we can find a variable
${\bm{\alpha}}^{\star}$ such that
$P({\bm{w}}({\bm{\alpha}}^{\star}))=D({\bm{\alpha}}^{\star})$, then, from the
strong duality, ${\bm{\alpha}}^{\star}$ and ${\bm{w}}({\bm{\alpha}}^{\star})$
can be recognized as optimal solutions to the dual problem (2) and the primal
problem (1) respectively. This paper aims to design a distributed algorithm
that efficiently solves (2) while considering the information of imbalanced
data, with the goal of mitigating the effect caused by data imbalance.

Fig. 1: Illustration of a tree-structured network, which has two layers. In
the network, a central station (root node) has three direct child nodes, i.e.,
sub-central node, denoted by $S$. Each sub-central node $S$ has three direct
child nodes, i.e., local workers, denoted by $W$.

##  3 Review of Distributed Algorithms with Balanced or Imbalanced Data

In previous studies, many researchers have addressed the issue of handling
balanced or imbalanced data in the design of distributed algorithms for ML
operations. For example, the authors in [22, 23] used the Alternating
Direction Method of Multipliers (ADMM) technique to deal with various types of
imbalanced data on a star network for classification problems. The authors in
[24] considered a separable optimization problem on a star network with
imbalanced data in the case when the number of local workers is larger than
the number of data points. Unfortunately, most of the research focused on
designing distributed algorithms on a star network, which is a very simple and
special network topology.

To deal with different types of network topologies, the researchers in [12,
13] extended the Distributed Dual Coordinate Ascent method to a general tree
network with $p$ layers (DDCA-Tree), where the root node is located in the
0-th layer and local workers are located in the $p$-th layer. Under the
assumption that every node on the network has balanced data, the design of the
distributed algorithm was studied. However, the design of an efficient
distributed algorithm that considers imbalanced data in a general tree network
has not been fully studied. This paper aims to address this research gap and
investigate this topic.

##  4 Generalized distributed dual coordinate ascent in a tree network

We aim to investigate the effect of imbalanced data on the convergence rate of
DDCA-Tree when applied to a general tree network. We will also propose an
enhanced version of DDCA-Tree that takes into account the imbalanced data to
improve its performance. Specifically, we will address the following
questions: (Q1) How does the convergence rate of DDCA-Tree differ when applied
to imbalanced data? (Q2) How can we design DDCA-Tree to better handle
imbalanced data by incorporating information about the imbalance?

To mitigate the effect of imbalanced data distribution, we can consider the
information on the imbalanced data distribution in the accumulation of global
parameters, which is the concept of Generalized distributed Dual Coordinate
Ascent on a general tree network (GDCA-Tree). In GDCA-Tree, described in
Algorithm 1, we propose to use different weights for global parameters that
local workers have in updating a global parameter at the $t$-th iteration,
${\bm{w}}^{(t)}$. More specifically, in the accumulation of the global
parameters, by considering the information of imbalanced data, we use a
weighted sum updating scheme for global parameter ${\bm{w}}^{(t)}$ in a
general tree node $Q$ having $K$ child nodes as

$\displaystyle{\bm{w}}^{(t)}={\bm{w}}^{(t-1)}+\sum_{k=1}^{K}\beta_{k}\bigtriangleup{\bm{w}}_{k},$
(3)

where $\beta_{k}$ is a weight considering imbalanced data factor for the
$k$-th updating parameter $\bigtriangleup{\bm{w}}_{k}$,
$\sum_{k=1}^{K}\beta_{k}=1$, $0\leq\beta_{k}\leq 1$.

For the convergence analysis of GDCA-Tree, let us define the local sub-
optimality gap, $\epsilon_{Q,k}$, at the $k$-th direct child node of a general
tree node $Q$ as follows:

$\displaystyle\epsilon_{Q,k}({\bm{\alpha}})\triangleq\underset{\hat{{\bm{\alpha}}}_{[Q;k]}}{\text{maximize}}\;D({\bm{\alpha}}_{[Q;1]},...,\hat{{\bm{\alpha}}}_{[Q;k]},...,{\bm{\alpha}}_{[Q;K]},{\bm{\alpha}}_{\overline{Q}})$
$\displaystyle\quad\quad\quad\quad\quad\quad-D({\bm{\alpha}}_{[Q;1]},...,{\bm{\alpha}}_{[Q;k]},...,{\bm{\alpha}}_{[Q;K]},{\bm{\alpha}}_{\overline{Q}}),$
(4)

where $[Q;k]$ is the set of indices of data points in the $k$-th direct child
node of $Q$, ${\bm{\alpha}}_{[Q;k]}$ is the partial vector of dual vector
${\bm{\alpha}}\in{\mathbb{R}}^{m}$ that corresponds to the data points that
the $k$-th direct child node of $Q$ has, and $\overline{Q}$ is complement of
an index set $Q$. This local sub-optimality gap represents the maximum
objective value gap that the $k$-th direct child node of $Q$ can achieve from
the current ${\bm{\alpha}}$ value. With this definition, we introduce the
following assumption.

######  Assumption 4.1 (Geometric improvement of GDCA-Tree at a direct child
node).

For a tree node $Q$ on the $i$-th layer, we assume that for any given
${\bm{\alpha}}$ at the $k$-th direct child node of $Q$, the GDCA-Tree provides
an update $\bigtriangleup{\bm{\alpha}}_{[Q;k]}$ such that

$\displaystyle{\mathbb{E}}[\epsilon_{Q,k}({\bm{\alpha}}_{[Q;1]},...,{\bm{\alpha}}_{[Q;k-1]},{\bm{\alpha}}_{[Q;k]}+\bigtriangleup{\bm{\alpha}}_{[Q;k]},...,{\bm{\alpha}}_{[Q;K]},{\bm{\alpha}}_{\overline{Q}})]$
$\displaystyle\leq\Theta_{i+1}\cdot\epsilon_{Q,k}({\bm{\alpha}}),$ (5)

where $\Theta_{i+1}\in[0,1)$ is local improvement.

It represents that if this assumption holds, then, the expectation of the
local sub-optimality gap at the $k$-th direct child node of a tree node $Q$ is
decreased by the factor of local improvement $\Theta_{i+1}$. When using Local
Stochastic Dual Coordinate Ascent (LocalSDCA), with a local dataset in a leaf
node in the $p$-th layer of a tree network, it is possible to achieve the
following proposition concerning the parameter $\Theta_{i+1}$:

######  Proposition 4.2.

([8] ) Assume that loss functions $\ell_{i}(\cdot)$ are $1/\gamma$-smooth. For
a tree node $Q$ on the $(p-1)$-th layer, its direct child node is a leaf node.
Then, for the leaf node $B$ in the $p$-th layer using LocalSDCA, Assumption
4.1 holds with

$\displaystyle\Theta_{p}=(1-\frac{\lambda m\gamma}{1+\lambda
m\gamma}\cdot\frac{1}{m_{B}})^{T_{p}},$ (6)

where $m_{B}$ is the size of data points that the leaf node $B$ has, and
$T_{p}$ is the number of iterations in LocalSDCA.

Input: $T_{i}\geq 1$, ${\bm{\alpha}}_{Q}$, ${\bm{w}}$

Initialization: ${\bm{\alpha}}_{[Q;k]}^{(0)}\leftarrow{\bm{\alpha}}_{[Q;k]}$
for all direct child nodes $k$ of node $Q$ ,
${\bm{w}}^{(0)}\leftarrow{\bm{w}}$

for  _ $t=1$ to $T_{i}$ _ do

for  _all direct child nodes $k=1,2,...,K_{i}$ of $Q$ in parallel_ do

$(\bigtriangleup{\bm{\alpha}}_{[Q;k]},\bigtriangleup{\bm{w}}_{k})$
$\leftarrow$ GDCA-Tree(${\bm{\alpha}}_{[Q;k]}^{(t-1)},{\bm{w}}^{(t-1)}$)

${\bm{\alpha}}_{[Q;k]}^{(t)}$ $\leftarrow$
${\bm{\alpha}}_{[Q;k]}^{(t-1)}+\beta_{k}\bigtriangleup{\bm{\alpha}}_{[Q;k]}$

end for

${\bm{w}}^{(t)}$ $\leftarrow$
${\bm{w}}^{(t-1)}+\sum_{k=1}^{K_{i}}\beta_{k}\bigtriangleup{\bm{w}}_{k}$

end for

Output:
$\bigtriangleup{\bm{\alpha}}_{Q}\triangleq{\bm{\alpha}}_{{Q}}^{(T_{i})}-{\bm{\alpha}}_{{Q}}^{(0)}$,
and
$\bigtriangleup{\bm{w}}_{Q}\triangleq{\bm{A}}_{Q}\bigtriangleup{\bm{\alpha}}_{Q}$

Algorithm 1 Generalized distributed Dual Coordinate Ascent in a general tree
node $Q$ (GDCA-Tree) on the layer-$i$, $i=1,2,...,p-1$

[t]  P(). Generalized Distributed Dual Coordinate Ascent (GDCA-Tree) for a
leaf node $Q$ on the layer-$p$ Input: $T_{p}\geq 1$,
${\bm{\alpha}}_{Q}\in{\mathbb{R}}^{|Q|}$, and ${\bm{w}}\in{\mathbb{R}}^{d}$
consistent with other coordinate blocks of ${\bm{\alpha}}$ s.t.
${\bm{w}}={\bm{A}}{\bm{\alpha}}$ Data: ${\\{({\bm{x}}_{i},y_{i})\\}}_{i\in Q}$
Initialization: $\bigtriangleup{\bm{\alpha}}_{Q}\leftarrow
0\in{\mathbb{R}}^{|Q|}$, and ${\bm{w}}^{(0)}\leftarrow{\bm{w}}$ for  _ $h=1$
to $T_{p}$ _ do

choose $i\in Q$ uniformly at random find $\bigtriangleup\alpha$ maximizing
$-\frac{\lambda m}{2}||{\bm{w}}^{(h-1)}+\frac{1}{\lambda
m}\bigtriangleup\alpha{\bm{x}}_{i}||^{2}-\ell_{i}^{*}(-(\alpha_{i}^{(h-1)}+\bigtriangleup\alpha))$
$\alpha_{i}^{(h)}$ $\leftarrow$ $\alpha_{i}^{(h-1)}+\bigtriangleup\alpha$
$(\bigtriangleup{\bm{\alpha}}_{Q})_{i}$ $\leftarrow$
$(\bigtriangleup{\bm{\alpha}}_{Q})_{i}+\bigtriangleup\alpha$ ${\bm{w}}^{(h)}$
$\leftarrow$ ${\bm{w}}^{(h-1)}+\frac{1}{\lambda
m}\bigtriangleup\alpha{\bm{x}}_{i}$

end for

Output: $\bigtriangleup{\bm{\alpha}}_{Q}$ and
$\bigtriangleup{\bm{w}}_{Q}\triangleq{\bm{A}}_{Q}\bigtriangleup{\bm{\alpha}}_{Q}$

From Proposition 4.2 and Assumption 4.1, over randomly chosen data points in
the LocalSDCA at a leaf node, we can have guaranteed improvement at its
parent’s node in terms of the expectation of local sub-optimality gap.

For Algorithm 1, we have the following convergence theorem at a general tree
node $Q$:

######  Theorem 4.3.

For a tree node $Q$ on the $i$-th layer, $i=0,...,p-1$, having $K_{i}$ direct
child nodes satisfying Assumption 4.1, with parameters $\Theta^{1}_{i+1}$,
$\Theta_{i+1}^{2}$, …, and $\Theta^{K_{i}}_{i+1}$. Assume that loss functions
$\ell_{i}(\cdot)$’s are $\nicefrac{{1}}{{\gamma}}$-smooth. Then for any input
${\bm{w}}$ to Algorithm 1 with $T_{i}$ iterations, the following geometric
convergence rate holds for $Q$:

$\displaystyle{\mathbb{E}}[D({\bm{\alpha}}_{Q}^{*},{\bm{\alpha}}_{\overline{Q}})-D({\bm{\alpha}}_{Q}^{(T_{i})},{\bm{\alpha}}_{\overline{Q}})]$
(7)
$\displaystyle\leq\underbrace{\bigg{(}\max_{k=1,...,K_{i}}\big{(}1-(1-\Theta^{k}_{i+1})\beta_{k}\big{)}\frac{\lambda
m\gamma}{\rho_{i}+\lambda m\gamma}\bigg{)}^{T_{i}}}_{\text{Convergence
bound}}$
$\displaystyle\quad\quad\quad\times\big{(}D(\bm{\alpha}_{Q}^{*},\bm{\alpha}_{\overline{Q}})-D({\bm{\alpha}}_{Q}^{(0)},{\bm{\alpha}}_{\overline{Q}})\big{)},$

where $\rho_{i}$ is any real number larger than $\rho_{min}$ defined as

$\displaystyle\rho_{min}\triangleq\underset{{\bm{\alpha}}_{Q}\in\mathbb{R}^{|Q|}}{\text{maximize}}\;\lambda^{2}m^{2}\frac{\sum_{k=1}^{K_{i}}\|{\bm{A}}_{[Q;k]}{\bm{\alpha}}_{[Q;k]}\|_{2}^{2}-\|{\bm{A}}_{Q}{\bm{\alpha}}_{Q}\|_{2}^{2}}{\|{\bm{\alpha}}_{Q}\|_{2}^{2}}\geq
0.$

###### Proof.

The dual objective value for a tree node $Q$ that has $K$ direct child nodes
is bounded by the following equation:

$\displaystyle
D\big{(}{\bm{\alpha}}^{(t+1)}_{[1:K]},{\bm{\alpha}}_{\overline{Q}}\big{)}$
$\displaystyle=D\big{(}{\bm{\alpha}}^{(t)}_{[1:K]}+\sum_{k=1}^{K}\beta_{k}\bigtriangleup{\bm{\alpha}}_{<[k]>},{\bm{\alpha}}_{\overline{Q}}\big{)}$
$\displaystyle\geq\sum_{k=1}^{K}\beta_{k}D\big{(}{\bm{\alpha}}^{(t)}_{[1:K]}+\bigtriangleup{\bm{\alpha}}_{<[k]>},{\bm{\alpha}}_{\overline{Q}}\big{)},$
(8)

where ${\bm{\alpha}}_{<[k]>}$ is ${\bm{\alpha}}_{[k]}$ with zero-padding to
increase the dimension, $Q=[1:K]=\cup_{k=1}^{K}[k]$ is the index set
corresponding to the node $Q$, and ${\bm{\alpha}}_{\overline{Q}}$ is the un-
updated coordinate. The inequality in (8) is obtained by using Jensen’s
inequality. Then, we have

$\displaystyle{\mathbb{E}}\bigg{[}D\big{(}{\bm{\alpha}}^{(t+1)}_{[1:K]},{\bm{\alpha}}_{\overline{Q}}\big{)}-D\big{(}{\bm{\alpha}}^{(t)}_{[1:K]},{\bm{\alpha}}_{\overline{Q}}\big{)}\bigg{]}$
$\displaystyle\geq{\mathbb{E}}\bigg{[}\sum_{k=1}^{K}\beta_{k}\big{(}D\big{(}{\bm{\alpha}}^{(t)}_{[1:K]}+\bigtriangleup{\bm{\alpha}}_{<[k]>},{\bm{\alpha}}_{\overline{Q}}\big{)}-D\big{(}{\bm{\alpha}}^{(t)}_{[1:K]},{\bm{\alpha}}_{\overline{Q}}\big{)}\big{)}\bigg{]}$
$\displaystyle={\mathbb{E}}\bigg{[}\sum_{k=1}^{K}\beta_{k}\big{(}\epsilon_{Q,k}({\bm{\alpha}}^{(t)}_{[1:K]},{\bm{\alpha}}_{\overline{Q}})-\epsilon_{Q,k}({\bm{\alpha}}^{(t)}_{[1:K]}+\bigtriangleup{\bm{\alpha}}_{<[k]>},{\bm{\alpha}}_{\overline{Q}})\big{)}\bigg{]}$
$\displaystyle=\sum_{k=1}^{K}\beta_{k}\big{(}{\mathbb{E}}[\epsilon_{Q,k}({\bm{\alpha}}^{(t)}_{[1:K]},{\bm{\alpha}}_{\overline{Q}})]-{\mathbb{E}}[\epsilon_{Q,k}({\bm{\alpha}}^{(t)}_{[1:K]}+\bigtriangleup{\bm{\alpha}}_{<[k]>},{\bm{\alpha}}_{\overline{Q}})]\big{)}$
$\displaystyle\geq\sum_{k=1}^{K}\beta_{k}(1-\Theta^{k})\big{(}\epsilon_{Q,k}({\bm{\alpha}}^{(t)}_{[1:K]},{\bm{\alpha}}_{\overline{Q}})\big{)}$
$\displaystyle\geq\bigg{(}\min_{k=1,2,...,K}\underbrace{(1-\Theta^{k})\beta_{k}}_{\begin{subarray}{c}\text{imbalanced
data factor}\\\ {\text{and its compensation
parameter}}\end{subarray}}\bigg{)}\underbrace{\sum_{k=1}^{K}\big{(}\epsilon_{Q,k}({\bm{\alpha}}^{(t)}_{[1:K]},{\bm{\alpha}}_{\overline{Q}})\big{)}}_{(A)}$

The lower-bound of $(A)$ can be obtained in Appendix A of [13], which leads to
(7). Due to space limitation, we omit the remaining proof. ∎

This theorem indicates that for a tree node $Q$, if its child nodes satisfy
the local sub-optimality gap with local improvement, then, at the node $Q$,
GDCA-Tree can have the convergence rate shown in (7). From Proposition 4.2, we
know that Assumption 4.1 holds at leaf nodes using LocalSDCA. Then, for the
convergence of GDCA-Tree in a general tree network, the left hand side of (7)
can be considered as the local sub-optimality gap that the tree node $Q$ can
achieve from $({\bm{\alpha}}^{(T_{i})}_{Q},{\bm{\alpha}}_{\overline{Q}})$, and
the convergence bound in (7) can be thought of as local improvement $\Theta$
from the parent node of $Q$. In this way, the convergence of GDCA-Tree in a
whole tree network can be understood in a recursive manner.

###  4.1 Impact of imbalanced data on the convergence speed

From Proposition 4.2, among all leaf nodes, with the same number of local
iterations, $T_{p}$, in LocalSDCA, the leaf node having the largest number of
data points will have the largest $\Theta_{p}$ (i.e., close to 1) due to
$\nicefrac{{1}}{{m_{B}}}$ term in (6), and become a bottleneck in the
convergence speed. Thus, $\nicefrac{{1}}{{m_{B}}}$ in (6) (or broadly
$(1-\Theta^{k})$) can be thought of as imbalanced data factor in a leaf node,
while $\beta_{k}$ can be considered as a compensation parameter for the
imbalanced data factor at the $k$-th leaf node by providing more weights on a
bottleneck node. Note that in the case of balanced data,
$\beta_{k}=\nicefrac{{1}}{{K}}$, $k=1,...,K$.

Furthermore, the parameter $\rho_{min}$ is a value indicating the similarity
of global parameter ${\bm{w}}$’s among the $K$ child nodes. If we have a
smaller $\rho$ value, i.e., smaller similarity between the global parameters
${\bm{w}}$ in child nodes, then we will have a larger convergence bound. In
other words, as the global parameters among child nodes become similar, the
convergence speed will decrease.

###  4.2 Determining the compensation parameter $\beta_{k}$ for imbalanced
data

In order to determine the weight parameter $\beta_{k}$ that compensates
imbalanced data effect, we consider the case where each node on a network has
different numbers of data points. Suppose we have a general node $Q$ which has
$K$ direct child nodes. The $k$-th child node has a partial dataset, where the
set of indices of data points is denoted by $[Q;k]$. Due to the scenario that
we consider here, the cardinality of the set $[Q;k]$, i.e., the number of data
points in the $k$-th direct child node of $Q$, is different from each other.
By considering the imbalanced numbers of data scenario, we propose the
following weight calculation for $\beta_{k}$ in the weighted sum updating
scheme (3):

$\displaystyle\beta_{k}=\nicefrac{{|[Q;k]|}}{{|Q|}},$ (9)

where the cardinality of the set of indices of data points in the $k$-th
direct child node of $Q$ is represented by $|[Q;k]|$. The weighted sum
updating scheme is used to put more weight on local updating parameters
obtained from processing more data. The intuition behind this is that if one
local worker (e.g., $W_{A}$) has most data and other workers (e.g., $W_{B}$)
have only a few, then, the solution obtained from $W_{A}$ is prone to be
closer to the global solution ${\bm{w}}^{\star}$. Thus, the global parameter
${\bm{w}}^{(t)}$ is updated based on the portion of data in each local worker
or sub-central node. The computation of $\beta_{k}$ can be done as a
preprocessing step and is negligible when compared to the coordinate ascent
operation.

###  4.3 Number of iterations considering imbalanced data

The convergence bound in (7) is a function of the number of iterations as well
as the imbalanced data factor. Therefore, the question raised is that is there
any relationship between the number of local iterations and the imbalanced
data? Let us answer this question here. For clarity and simplicity, we
consider a cluster shown in Fig. 1, where the sub-central node is denoted by
$Q$. For the optimal number of local iterations to have minimum execution time
in convergence, as in [13], we consider the following optimization problem:

$\displaystyle\underset{T_{p}\geq
0}{\text{minimize}}\;\bigg{(}1-(1-\Theta^{k}_{p})\beta_{k}\frac{\lambda
m\gamma}{\rho_{i}+\lambda m\gamma}\bigg{)}^{T_{p-1}},$ (10)

where $T_{p}$ and $T_{p-1}$ are the numbers of local iterations in a leaf node
and its parent node respectively. $\Theta_{p}$ is introduced in (6). Since the
total time, $t_{total}$, for distributed ML process in a cluster can be
expressed as $t_{total}=(t_{lp}T_{p}+t_{delay}+t_{cp})\cdot T_{p-1}$, where
$t_{lp}$ is local processing time per one iteration in a leaf node,
$t_{delay}$ is time for communication delay in round trip between a leaf node
and its parent node, i.e., sub-central node in Fig. 1, and $t_{cp}$ is time
for accumulation at a sub-central node. Then, by plugging (6) into (10),
denoting $\frac{\lambda m\gamma}{1+\lambda m\gamma}$ as $c_{1}$, and
$\frac{\lambda m\gamma}{\rho_{i}+\lambda m\gamma}$ as $c_{2}$, we have

$\displaystyle\underset{T_{p}\geq
0}{\text{minimize}}\;\bigg{(}1-(1-(1-\frac{c_{1}}{|[Q;k]|})^{T_{p}})c_{2}\cdot\frac{|[Q;k]|}{|Q|}\bigg{)}^{\frac{t_{total}}{t_{lp}T_{p}+t_{delay}+t_{cp}}}.$

From [13], the optimal number of local iterations $T_{p}$ at the leaf node can
be obtained as follows:

$\displaystyle
T_{p}=\frac{1}{\ln(1-\frac{c_{1}}{|[Q;k]|})}W\bigg{(}(1-\frac{c_{1}}{|[Q;k]|})^{r}\ln(1-c_{2}\cdot\frac{|[Q;k]|}{|Q|})\bigg{)}-r,$

where $c_{1},c_{2}\in[0,1)$, communication delay severity level between the
local processing time and the communcation delay is denoted by $r$, i.e.,
$r=\nicefrac{{(t_{delay}+t_{cp})}}{{t_{lp}}}$, and $W(\cdot)$ is the Lambert
W-function [25]. If the delay severity $r$ is zero or small enough to ignore,
then, the Lambert W-function term can be expressed as a small negative
constant, and $\ln(1-x)\approx-x$ for small $x$, we have

$\displaystyle T_{p}\propto|[Q;k]|.$ (11)

Namely, the number of local iterations need to be increased proportionally to
the number of data points in a node in order to minimize the execution time
during convergence. Therefore, by taking into account the size of the data in
the bottleneck node, we propose the delayed GDCA-Tree, where the number of
local iterations is determined by the size of the local dataset in a
bottleneck node. In the delayed GDCA-Tree method, the sharing of information
between local workers and the sub-central node is delayed, but with a larger
number of local iterations. This results in faster convergence speed.

##  5 Numerical Experiments

To validate the performance of the delayed GDCA-Tree, we conduct simulations
and compare it to the standard DDCA-Tree when dealing with imbalanced data on
a tree network. We test the method on various machine learning tasks including
regression and classification using the following datasets: wine quality
dataset111https://archive.ics.uci.edu/ml/datasets/wine+quality and Covtype
dataset222https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html#covtype.binary.

In the wine quality dataset, there are 6493 data points with 12 attributes.
The twelfth attribute, quality, is used as the measurement ${\bm{y}}$. Each
instance is normalized with $\ell_{2}$ norm to ensure
$\|{\bm{x}}_{i}\|_{2}\leq 1$, $i=1,...,m$. To distribute the dataset over a
tree network, we organize a two-layered tree network with one central node,
two sub-central nodes (denoted by $S_{1}$ and $S_{2}$), and four local workers
(denoted by $W_{i}$, $i=1,...,4$). Each sub-central node has two local
workers. The tuning parameter $\lambda$ in (2) is set to 1. We consider the
case where each local worker has a different number of data points. To
simulate this scenario, we unevenly distribute the wine quality data into four
local workers. 30% of the total dataset are allocated to three local workers,
$W_{1}$, $W_{2}$, and $W_{3}$ (10% each, which is 649 data points). The
remaining 70% of the total dataset (4546 data points) is allocated to one
local worker, i.e., $W_{4}$, without overlapping the data among local workers.

In the delayed GDCA-Tree, we use (9) to calculate $\beta_{k}$ for each local
worker. Therefore, for each updating global parameter
$\bigtriangleup{\bm{w}}_{k}$ that each local worker has from its local
dataset, $\beta_{1}=\beta_{2}=0.5(=649/(649+649))$,
$\beta_{3}=0.1249(=649/(649+4546))$, and $\beta_{4}=0.8752(=4546/(649+4546))$.
At the sub-central nodes $S_{1}$ and $S_{2}$, $0.2(=(649+649)/6493)$ and
$0.8(=(649+4546)/6493)$ are used for $\beta_{k}$ respectively.

In order to obtain statistical results, we run 100 random trials. From the 100
trials, we calculate the average execution time at each outer iteration and
compute the average duality gap
$P({\bm{w}}({\bm{\alpha}}^{(t)}))-D({\bm{\alpha}}^{(t)})$ at the central
station. For the number of local iterations $T_{p}$ in Procedure 4, we use
$T_{p}=100$. And for delayed GDCA-Tree, we use 300 local iterations for
$T_{p}$ by considering the increased number of data points in $W_{4}$. As
shown in Fig. 2(a), the delayed GDCA-Tree (red solid line) can improve the
convergence speed, compared to the standard DDCA-Tree [13] with imbalanced
data (blue dotted line). The figure illustrates that by using the delayed
GDCA-Tree, we can mitigate the effect of imbalanced data on distributed dual
coordinate ascent in a tree network.

(a) Wine quality

(b) Covtype

Fig. 2: Comparison between standard DDCA-Tree (blue) and delayed GDCA-Tree
(red) with imbalanced data.

Additionally, we run binary classification with Covtype dataset having a total
of 581012 instances and 12 attributes. The dataset is normalized and the
labels are set to be in the set {-1,1}. A standard hinge loss and $\ell_{2}$
regularization are used for a linear SVM. The tree network used has one
central node, two sub-central nodes, and eight local workers, where each sub-
central node has four local workers. The data is distributed in an imbalanced
way, with 5% of the total dataset going to seven local workers, and 65% going
to one local worker with no overlap. The number of communications between the
local workers and the sub-central node is set to 10. The numbers of local
iterations in local workers in the standard DDCA-Tree and the delayed GDCA-
Tree are set to 1000 and 4000 respectively. Fig. 2(b) demonstrates that under
the imbalanced data scenario, the delayed GDCA-Tree can improve the
convergence speed of the DDCA-Tree by considering the information of
imbalanced data.

## References

  * [1] M. Teng and F. Wood,  “Bayesian distributed stochastic gradient descent,”  in Proceedings of Advances in Neural Information Processing Systems, 2018, vol. 31.
  * [2] N. Ferdinand and S. C. Draper,  “Anytime stochastic gradient descent: A time to hear from all the workers,”  in Proceedings of Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 2018, pp. 552–559.
  * [3] G. Cong, O. Bhardwaj, and M. Feng,  “An efficient, distributed stochastic gradient descent algorithm for deep-learning applications,”  in Proceedings of International Conference on Parallel Processing. IEEE, 2017, pp. 11–20.
  * [4] S. Shi, Q. Wang, X. Chu, and B. Li,  “A DAG model of synchronous stochastic gradient descent in distributed deep learning,”  in Proceedings of IEEE International Conference on Parallel and Distributed Systems. IEEE, 2018, pp. 425–432.
  * [5] N. Ferdinand, B. Gharachorloo, and S. C. Draper,  “Anytime exploitation of stragglers in synchronous stochastic gradient descent,”  in Proceedings of IEEE International Conference on Machine Learning and Applications. IEEE, 2017, pp. 141–146.
  * [6] C.-J. Hsieh, K.-W. Chang, C.-J. Lin, S. S. Keerthi, and S. Sundararajan,  “A dual coordinate descent method for large-scale linear SVM,”  in Proceedings of International Conference on Machine Learning. ACM, 2008, pp. 408–415.
  * [7] T. Yang,  “Trading computation for communication: Distributed stochastic dual coordinate ascent,”  in Proceedings of Advances in Neural Information Processing Systems, 2013, pp. 629–637.
  * [8] M. Jaggi, V. Smith, M. Takác, J. Terhorst, S. Krishnan, T. Hofmann, and M. I. Jordan,  “Communication-efficient distributed dual coordinate ascent,”  in Proceedings of Advances in Neural Information Processing Systems, 2014, pp. 3068–3076.
  * [9] S. Shalev-Shwartz and T. Zhang,  “Stochastic dual coordinate ascent methods for regularized loss minimization,”  Journal of Machine Learning Research, vol. 14, pp. 567–599, 2013\.
  * [10] Y. Deng and M. Mahdavi,  “Local stochastic gradient descent ascent: Convergence analysis and communication efficiency,”  in Proceedings of International Conference on Artificial Intelligence and Statistics. PMLR, 2021, pp. 1387–1395.
  * [11] A. Devarakonda, K. Fountoulakis, J. Demmel, and M. W. Mahoney,  “Avoiding communication in primal and dual block coordinate descent methods,”  SIAM Journal on Scientific Computing, vol. 41, no. 1, pp. C1–C27, 2019.
  * [12] M. Cho, L. Lai, and W. Xu,  “Generalized distributed dual coordinate ascent in a tree network for machine learning,”  in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2019, pp. 3512–3516.
  * [13] M. Cho, L. Lai, and W. Xu,  “Distributed dual coordinate ascent in general tree networks and communication network effect on synchronous machine learning,”  IEEE Journal on Selected Areas in Communications, vol. 39, no. 7, pp. 2105–2119, 2021.
  * [14] S.-Y. Zhao and W.-J. Li,  “Fast asynchronous parallel stochastic gradient descent: A lock-free approach with convergence guarantee,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2016, pp. 2379–2385.
  * [15] R. Zhang, S. Zheng, and J. T. Kwok,  “Fast distributed asynchronous SGD with variance reduction,”  CoRR, abs/1508.01633, 2015.
  * [16] X. Lian, W. Zhang, C. Zhang, and J. Liu,  “Asynchronous decentralized parallel stochastic gradient descent,”  in Proceedings of International Conference on Machine Learning (ICML). PMLR, 2018, pp. 3043–3052.
  * [17] Z. Huo and H. Huang,  “Distributed asynchronous dual free stochastic dual coordinate ascent,”  in Proceedings of the IEEE International Conference on Data Mining, 2018.
  * [18] C.-J. Hsieh, H.-F. Yu, and I. S. Dhillon,  “PASSCoDe: Parallel asynchronous stochastic dual co-ordinate descent,”  in Proceedings of the International Conference on Machine Learning, 2015, vol. 15, pp. 2370–2379.
  * [19] J. Liu, S. Wright, C. Ré, V. Bittorf, and S. Sridhar,  “An asynchronous parallel stochastic coordinate descent algorithm,”  in Proceedings of International Conference on Machine Learning. PMLR, 2014, pp. 469–477.
  * [20] T. Sun, R. Hannah, and W. Yin,  “Asynchronous coordinate descent under more realistic assumptions,”  in Proceedings of Advances in Neural Information Processing Systems, 2017, vol. 30.
  * [21] S. Boyd and L. Vandenberghe,  Convex optimization,  Cambridge university press, 2004.
  * [22] H. Wang, Y. Gao, Y. Shi, and H. Wang,  “A fast distributed classification algorithm for large-scale imbalanced data,”  in Proceedings of IEEE International Conference on Data Mining. IEEE, 2016, pp. 1251–1256.
  * [23] H. Wang, M. Xiao, C. Wu, and J. Zhang,  “Distributed classification for imbalanced big data in distributed environments,”  Wireless Networks, pp. 1–12, 2021.
  * [24] J. Konečnỳ, B. McMahan, and D. Ramage,  “Federated optimization: Distributed optimization beyond the datacenter,”  arXiv preprint arXiv:1511.03575, 2015.
  * [25] R. M. Corless, G. H. Gonnet, D. EG. Hare, D. J. Jeffrey, and D. E. Knuth,  “On the LambertW function,”  Advances in Computational Mathematics, vol. 5, no. 1, pp. 329–359, 1996.