Editing units of consumable required for each HSI

There are a few challenges with the way in which units of consumables required are coded in modules -

Inconsistency in coding the quantity of consumables needed - Some modules express the units needed in terms of the proportion of the unit of issue (package sold) as per the name of the consumable in the original consumable name list from the 2016 One Health Tool. For instance, see line 1040 of alri.py where 0.01 of a package is required per HSI. Others specify these in terms of the number of individual units (smallest divisible unit). For instance, the contraception module assume 30 condoms issued per HSI.

Using  the unit of issue as defined in the 2016 One Health Tool might lead to errors in costing especially if these units of issue change over time and costs are expressed in terms of a different unit of issue. For eg., costs expressed in terms of $/pack of 30 tablets whereas module coded to provide quantities assuming a pack  of 60 tablets.

In order to address the inconsistency and ensure accuracy of costing, we wish to recode consumable quantities required by HSIs within modules to represent the quantity of the chosen unit of the consumable in terms of which price is expressed, such as a single vial, single ampoule, 1 gram of drug, 1ml of liquid etc. Some examples below -

Consumable_name_tlo
Chosen_unit_for_price
Final_price_per_chosen_unit   (USD, 2023)

Gentamycin,   injection, 40 mg/ml in 2 ml vial
2 ml vial (40 mg/ml)
0.053693068

Ciprofloxacin   250mg_100_CMST
1 mg
0.000058701

Ferrous   sulphate 200mg _1000_CMST
1 tablet (Ferrous   sulphate 200mg/Folic acid 25mg)
0.001982148

Levonorgestrel   0.0375 mg, cycle
Tablet   (Levonorgestrel 0.0375 mg/~30 micrograms)
0.043909897

Injectable   artesunate
1 ampoule (60 mg   artesunate)
0.109717557

Doxorubicin,   50 mg vial
1 vial (50 mg)
19.308152109

Bandage, crepe   7.5cm x 1.4m long , when stretched
1 cm long (7.5 cm   wide)
0.002716640

For this PR, we request module leads to express the quantities of consumables needed in terms of theChosen_unit_for_price column of the consumables sheet in ~resources/costing/ResourceFile_cost_consumables.xlsx. If you think the unit in the resource file needs to be changed, feel free to suggest this.
Modules (Module leads to mark as done once updated):

[ ] ALRI
[ ] Bladder cancer
[ ] Breast Cancer
[ ] Cardiometabolic disorders
[ ] Care of women during pregnancy
[ ] Contraception
[ ] COPD
[ ] Depression
[ ] Diarrhea
[ ] EPI
[ ] Epilepsy
[x] HIV
[ ] Labour
[x] Malaria
[ ] Measles
[ ] Oesophagal cancer
[ ] Other adult cancers
[ ] Prostate cancer
[ ] Road traffic injuries
[ ] Schisto
[ ] Stunting
[ ] TB
[ ] Wasting

Useful references:

WHO Model list of essential medicines
Health Action International Price and Avilability List  - useful international database but incomplete
Live Malawi CMST Catalogue

Hi @tdm32 @joehcollins @EvaJanouskova @mnjowe @andrew-phillips-1 @timcolbourn @tbhallett
As discussed at the quarterly meeting, this is the PR to update the quantity of consumables requested within disease modules. I have now updated the consumables cost data to express costs in terms of smaller units, as described in the message above. Hope the request makes sense - feel free to reach out if not and I'm happy to clarify here or on a call.

@tdm32 There are a few HIV consumables for which I wasn't able to extract costs from the sources that I was using (CMST Catalogue, EHP Tool, 2016 One Health Tool). I think we can use the Global Fund Pooled Purchasing Mechanism Reference Prices. Would you have any thoughts on what would be the appropriate regimen for the following three item codes -
2671 | First-line ART   regimen: adult
2672 | First line ART   regimen:  child
2673 | First line ART regimen: young child
199 | Nevirapine, tablet,   200 mg - only the cost of oral solution ("Nevirapine 10mg/mL oral suspension, 100mL bottle") is available
Similarly, for TB I'm using the Stop TB partnership Catalogue. Cat I and III Patient Kits (item codes 176 and 178) are listed here so those were straightforward to cost. But the following TB consumables aren't listed. Could you advise on these?
Note that for now I have included the cost as listed in the 2016 One Health Tool (but I had to use the unit 1 full treatment ("First line treatment for retreatment TB cases for adults") because the One Health Tool doesn't break down the cost.
184 | Solid culture   and DST
177 | Cat. II Patient Kit   A1
179 | Cat. II Patient Kit   A2
181 | Treatment:   second-line drugs
182 | MDR Fuel

@joehcollins @EvaJanouskova Should the following consumables be treated as equipment? I've currently included the cost in the spreadsheet based on recent Central Medical Stores data but given how consumables are costed, we'll end up overestimating the cost of these by counting it each time the item code is needed. Another option is that we divide the unit costs by the number of times we expect these devices can be used before needing to be discarded.
2669 | Forceps,   obstetric ($23)
2670 | Vacuum, obstetric ($129)
@joehcollins I also have a couple of notes on nutrition consumables if you'll be working on these. I extracted many of the costs from the UNICEF Supply Catalogue because UNICEF is the main supplier of nutrition consumables in Malawi and these were not in the Central Medical Stores list. Let me know if you think I should verify these with Ines -
213 | SAM theraputic   foods  |   Cost based on   'Therapeutic CMV,tin 800g/CAR-6' from the UNICEF Supply Catalogue
208 | Corn Soya Blend (or   Supercereal - CSB++)  |    Cost based on   'Emergency food ration,500g/CAR-24' from the UNICEF Supply Catalogue
214 | SAM medicines  |  Used 2016 One Health Tool cost but it's not clear what the consumable is
1015 | Dietary supplements   (country-specific)  |  Cost based on    'Multiple micrn. pdr,custom sach./PAC-30'  from the UNICEF Supply Catalogue

Thanks very much @sakshimohan  -- this all looks very good and reasonable.
So, the definitive source is in the file resources/costing/ResourceFile_Costing.xlsx: sheet Consumables, column Z (shown below too).
I am wondering if we should also remove the columns named Costs and Expected Units per Case from the ResourceFile_Consumables_Items_and_Packages.csv, to avoid people (in the future) looking in the wrong place for this information/ and potentially merging in this column Z for each item_code there...?

Item_Code
Consumable_name_tlo
Chosen_unit_for_price

0
Levonorgestrel 0.0375 mg, cycle
Tablet (Levonorgestrel 0.0375 mg/~30 micrograms)

1
Levonorgestrel 0.15 mg + Ethinyl estradiol 30 mcg (Microgynon), cycle //97.2% users//; or Levonorgestrel 0.0375 mg, cycle //2.8% users//
Tablet (Levonorgestrel 0.15 mg + Ethinyl estradiol 30 mcg)

2
Condom, male
Single condom

3
Depot-Medroxyprogesterone Acetate 150 mg - 3 monthly
1 ml vial

5
Povidone iodine, solution, 10 %, 5 ml per injection
1 ml

6
Syringe, Autodisable SoloShot IX
1 syringe

7
IUD, Copper T-380A
1 IUD

8
Lidocaine HCl (in dextrose 7.5%), ampoule 2 ml
2 ml ampoule

9
Syringe, needle + swab
1 syringe

11
Needle suture intestinal round bodied A circle trocar_6_CMST
1 needle suture

12
Jadelle (implant), box of 2_CMST //84.8% users//; or Implanon (Etonogestrel 68 mg) //15.2% users//
1 implant

13
Implanon (Etonogestrel 68 mg)
1 implant

14
Atropine sulphate, injection, 1 mg in 1 ml ampoule
1 ml ampoule (1 mg/ml)

15
Diazepam, injection, 5 mg/ml, in 2 ml ampoule
2 ml ampoule (5 mg/ml)

16
Lidocaine, injection, 1 % in 20 ml vial
1 ml (1%)

17
Lidocaine, spray, 10%, 500 ml bottle
1 ml (10%)

21
Tape, adhesive, 2.5 cm wide, zinc oxide, 5 m roll
1 cm long (2.5 cm wide)

23
Paracetamol, tablet, 500 mg
1 mg

24
Cotton swab
1 g

25
Female Condom_1_CMST
1 condom

26
Misoprostol, tablet, 200 mcg
1 mcg

27
Ampicillin, powder for injection, 500 mg, vial
1 mg

28
Gentamycin, injection, 40 mg/ml in 2 ml vial
2 ml vial (40 mg/ml)

29
Metronidazole, injection, 500 mg in 100 ml vial
100 ml vial for IV (5 mg/ml)

30
Tetracycline, tablet, 500 mg
1 mg

33
Sodium chloride, injectable solution, 0,9 %, 500 ml
1 ml (0.9%)

39
Tetanus toxoid, injection
5 ml injection

40
Blood collecting tube, 5 ml
5 ml tube

41
Test, Rapid plasma reagin (RPR)
1 test

42
Benzathine benzylpenicillin, powder for injection, 2.4 million IU
1 injection (1.44g)

44
Ferrous sulphate 200mg _1000_CMST
1 tablet (Ferrous sulphate 200mg/Folic acid 25mg)

47
Urine analysis
1 bottle

49
Cotton wool, 500g_Each_FF007800_CMST
1 g

50
Haemoglobin test (HB)
1 test

52
Albendazole, tablet, 400 mg
1 g

56
Oxytocin, injection, 10 IU in 1 ml ampoule
1 ml ampoule (10 IU/ml)

57
Bag, urine, collecting, 2000 ml
1 2000ml bag

58
Foley catheter
1 catheter

60
Hydralazine, powder for injection, 20 mg ampoule
1 ml (20 mg/ml)

61
Magnesium sulfate, injection, 500 mg/ml in 10-ml ampoule
10 ml ampoule (500 mg/ml)

64
Infant resuscitator, clear plastic + mask + bag_1_CMST
1 resuscitator

65
Ketamine, 10 ml vial, 50 mg/ml
10 ml vial (50 mg/ml)

66
Cefazolin, ampoule, 500 mg
1 mg

67
Blade, surgical, no. 22, sterile, disposable
1 blade

68
Tetracycline eye ointment, 1 %, tube 5 mg
1 g

70
Amoxicillin, powder/oral suspension, 125 mg/5 ml
1 ml (125mg/5ml)

71
Paracetamol syrup 120mg/5ml_0.0119047619047619_CMST
1 ml (120mg/5ml)

73
Iodine strong 10% solution_500ml_CMST
1 ml (10%)

74
Chlorhexidine 1.5% solution_5_CMST
1 ml (1.5%)

75
Gauze, swabs 8-ply 10cm x 10cm_100_FF010800_CMST
1 swab (10cm X 10cm)

76
Scalpel blade size 22 (individually wrapped)_100_CMST
1 blade

80
Giving set iv administration + needle 15 drops/ml_1_CMST
1 IV set

81
Ceftriaxone 1g, PFR_1_CMST
1 g

82
Catheter Foley's + urine bag (2000ml) 14g_1_CMST
1 catheter

84
Declofenac injection_1_CMST
3 ml ampoule (75 mg/ml)

86
Ampicillin injection 500mg, PFR_1_CMST
1 injection (500 mg)

88
surgical face mask, disp., with metal nose piece_50_IDA
1 mask

90
Halothane (fluothane)_250ml_CMST
1 ml

92
Metronidazole 200mg_1000_CMST
1 mg

93
Saline solution
1 ml (0.9%)

98
Water for injection, 10ml_1_CMST
10 ml ampoule

99
Benzylpenicillin 3g (5MU), PFR_1_CMST
1 g

101
Gauze, absorbent 90cm x 40m_1_CMST
1 cm long (84 cm wide)

102
Oxytocin 10 IU/ml, 1ml_1_CMST
1 ml (10 IU/ml)

103
Bandage, plaster of paris 15cm_12_CMST
1 bandage (15cm)

105
Metronidazole 5mg/ml, 100ml_1_CMST
1 ml (5 mg/ml)

106
Gentamicin Sulphate 40mg/ml, 2ml_1_CMST
2 ml vial (40 mg/ml)

112
Glove surgeons size 7 sterile_Pair_HH080400_CMST
1 glove

113
Paracetamol 500mg_1000_CMST
1 mg

117
Clean delivery kit
1 kit

120
Cannula iv (winged with injection pot) 16_1_CMST
1 cannula

123
Erythromycin, tablet, 250 mg
1 mg

125
Amoxycillin 250mg_1000_CMST
1 mg

126
Lancet, blood, disposable
1 lancet

127
Oxygen, 1000 liters, primarily with oxygen cylinders
1 litre

128
Complete blood count
1 bottle (5ml)

130
Gentamicin 40mg/ml, 2ml_1_CMST
2 ml vial (40 mg/ml)

134
Tube, feeding CH 8_1_CMST
1 tube

135
Disposables gloves, powder free, 100 pieces per box
1 glove

141
Blood, one unit
?

144
Hand washing with soap - cost per client
1ml

146
ORS, sachet
1 sachet

147
Zinc, tablet, 20 mg
1 mg

148
Ciprofloxacin 250mg_100_CMST
1 mg

149
ringer's lactate (Hartmann's solution), 1000 ml_12_IDA
1 ml

153
Syringe, autodisposable, BCG, 0.1 ml, with needle
1 dose

154
Safety box for used syringes/needles, 5 liter
1 box

155
Pneumococcal vaccine
1 dose

157
HPV vaccine
1 dose

162
Sulfamethoxazole + trimethropin, tablet 400 mg + 80 mg
1 ml (2.4 mg/ml)

163
Malaria test kit (RDT)
1 test

164
Lumefantrine 120mg/Artemether 20mg, 30x18_540_CMST
1 tablet (Lumefantrine 120mg/Artemether 20mg)

170
Injectable artesunate
1 ampoule (60 mg artesunate)

171
Cannula iv (winged with injection pot) 18_1_CMST
1 cannula

175
X-ray
1 film (35cm  X 43cm)

178
Cat.  I & III Patient Kit A
1 kit (6 blisters (168) 4-FDC tablets (R150/H75/Z400/E275); 12 blisters (336) 2-FDC tablets (R150/H75))

177
Cat. II Patient Kit A1
1 full treatment ("First line treatment for retreatment TB cases for adults")

179
Cat. II Patient Kit A2
1 full treatment ("First line treatment for retreatment TB cases for children")

181
Treatment: second-line drugs
1 full treatment ("Case management of MDR cases")

182
MDR Fuel
1 assay kit

200
DNA PCR for infant after birth
1 test

205
Nutrition supplements in first six months drugs/supplies to service a client
1 sachet (multiple micronutrient)

207
Smear test
1 test

208
Corn Soya Blend (or Supercereal - CSB++)
1 g

214
SAM medicines
1 full treatment ("Management of severe malnutrition (children)")

266
Leucovorin 50mg injection
1 vial (5 mg)

281
Thiamine (vitamin B1), 100 mg

283
Methadone, 5 mg
1 ml

285
Mectizan
1 full treatment ("Onchocerciasis mass drug administration")

287
I/V R/lactate 3Lts
1 full treatment ("LF hydrocele surgery")

290
Underpants
1 underpant

1015
Dietary supplements (country-specific)
1 sachet

1197
Pentavalent vaccine (DPT, Hep B, Hib)
1 dose

1221
Supplementary spread, sachet 92g/CAR-150
1 sachet (92g)

2580
vitamin K1 (phytomenadione) 1 mg/ml, 1 ml, inj._100_IDA
1 ml (2 mg/ml)

2671
First-line ART regimen: adult (30Kg+) ( ABC 600mg /  3TC 300mg)
?

2672
First line ART regimen:  child (larger weight-20-24.9kg) ( ABC 120mg /  3TC 60mg + DTG 50mg)
?

2673
First line ART regimen: young child (smaller weight-3.-19.9kg) ( ABC 120mg /  3TC 60mg + 10 DTG 10mg )
?

Also, here is the updated Module Leaders designations.

I am wondering if we should also remove the columns named Costs and Expected Units per Case from the ResourceFile_Consumables_Items_and_Packages.csv, to avoid people (in the future) looking in the wrong place for this information/ and potentially merging in this column Z for each item_code there...?

Makes sense, @tbhallett. I'll update generate_consumables_item_codes_and_packages.py remove those columns.

I am wondering if we should also remove the columns named Costs and Expected Units per Case from the ResourceFile_Consumables_Items_and_Packages.csv, to avoid people (in the future) looking in the wrong place for this information/ and potentially merging in this column Z for each item_code there...?

I do not agree with this. I always felt it would be much better if consumables in all modules are rather called by the Intervention_Pkg name than being defined inside the code. That way, it would be much easier to set up the model for other countries later. With all set within the model instead in the RF, it will be pain then. Costs could be in the separate RF per chosen unit, that is okay. But the names of Intervention_Pkg with the setup of consumables and expected units for that intervention would be better if they are in the ResourceFile_Consumables_Items_and_Packages.csv
In future, I would also like to see there a column saying which consumable is essential/optional for that Intervention_Pkg, so nothing has to be in the code and can potentially differ for different countries without touching the code.

Thanks for this @sakshimohan. Are you happy for us to just push any changes to this branch?

@joehcollins @EvaJanouskova Should the following consumables be treated as equipment? I've currently included the cost in the spreadsheet based on recent Central Medical Stores data but given how consumables are costed, we'll end up overestimating the cost of these by counting it each time the item code is needed. Another option is that we divide the unit costs by the number of times we expect these devices can be used before needing to be discarded.
2669 | Forceps, obstetric ($23) 2670 | Vacuum, obstetric ($129)
@joehcollins I also have a couple of notes on nutrition consumables if you'll be working on these. I extracted many of the costs from the UNICEF Supply Catalogue because UNICEF is the main supplier of nutrition consumables in Malawi and these were not in the Central Medical Stores list. Let me know if you think I should verify these with Ines -
213 | SAM theraputic foods | Cost based on 'Therapeutic CMV,tin 800g/CAR-6' from the UNICEF Supply Catalogue 208 | Corn Soya Blend (or Supercereal - CSB++) | Cost based on 'Emergency food ration,500g/CAR-24' from the UNICEF Supply Catalogue 214 | SAM medicines | Used 2016 One Health Tool cost but it's not clear what the consumable is 1015 | Dietary supplements (country-specific) | Cost based on 'Multiple micrn. pdr,custom sach./PAC-30' from the UNICEF Supply Catalogue

r.e. forceps/vacuum we currently have kept these items as consumables in the model so we can use their presumed availability to determine if the relevant intervention does or doesnt happen. This could maybe be replaced when we have the essential consumable equipment logic. Wonder what @tbhallett thinks about doing the cost per use approach? Otherwise we just ignore until we cost it properly with the equipment approach...
r.e. nutrition bits - i think they're stunting/wasting interventions so that would be @EvaJanouskova

Another quick question for @sakshimohan  - is there an error for item code 52 (Albendazole, tablet, 400mg) as the chosen unit for price is 1g when for item code 45 (Albendazole 200mg) its 1mg? Also maybe i've misunderstood but then in the code when i want an individual to have a single dose of item code 52, i'm requesting 400 of that item code? This varies from other items which have a smallest unit of one tablet?

Thanks for this @sakshimohan. Are you happy for us to just push any changes to this branch?

Hi @joehcollins. Yes, absolutely. Feel free to push changes to this branch.

r.e. forceps/vacuum we currently have kept these items as consumables in the model so we can use their presumed availability to determine if the relevant intervention does or doesnt happen. This could maybe be replaced when we have the essential consumable equipment logic. Wonder what @tbhallett thinks about doing the cost per use approach? Otherwise we just ignore until we cost it properly with the equipment approach...

Makes sense if you think it is important to account for their availability. I'll have a quick chat about this during my meeting with Tim this morning.

Another quick question for @sakshimohan - is there an error for item code 52 (Albendazole, tablet, 400mg) as the chosen unit for price is 1g when for item code 45 (Albendazole 200mg) its 1mg? Also maybe i've misunderstood but then in the code when i want an individual to have a single dose of item code 52, i'm requesting 400 of that item code? This varies from other items which have a smallest unit of one tablet?

Sorry that was indeed a typo - the price is expressed at $/mg. I've corrected this now. It's correct that for Albendazole you would need to request 400 units of item code 52 if you need one 400mg tablet. I realise that this can be confusing but we decided to go with this after discussing at the quarterly meeting that some of these tablets might be sold in different concentrations. For eg., albendazole itself is sold as either 200 mg or 400 mg tablets, even though you're using these item codes 45 and 52 for different interventions.
I've tried to use tablets only in  the cases where there were two compounds involved (eg. Ferrous sulphate/Folic Acid) because there wasn't an easy way to express these in terms of smaller quantities. Hope this makes sense. If, based on your medical knowledge, you think that some of these should be reverted to tablets because it makes logical sense, feel free to suggest this and I can make the update.

@sakshimohan - can we have the minimum unit for disposable gloves as 2 as opposed to 1 glove - so we dont need to specify we need two every time in the code...
Thanks!

Sure thanks, @joehcollins. I've updated the unit for items 112, 135 and 1933 to "1 pair of gloves".

@tdm32 There are a few HIV consumables for which I wasn't able to extract costs from the sources that I was using (CMST Catalogue, EHP Tool, 2016 One Health Tool). I think we can use the Global Fund Pooled Purchasing Mechanism Reference Prices. Another possible source is this CHAI ARV Reference Price list but this is older (2016). Would you have any thoughts on what would be the appropriate regimen for the following three item codes -
2671 | First-line ART regimen: adult 2672 | First line ART regimen:  child 2673 | First line ART regimen: young child
199 | Nevirapine, tablet, 200 mg - only the cost of oral solution ("Nevirapine 10mg/mL oral suspension, 100mL bottle") is available
Similarly, for TB I'm using the Stop TB partnership Catalogue (info on patient kits here). Cat I and III Patient Kits (item codes 176 and 178) are listed here so those were straightforward to cost. But the following TB consumables aren't listed. Could you advise on these? Note that for now I have included the cost as listed in the 2016 One Health Tool (but I had to use the unit 1 full treatment ("First line treatment for retreatment TB cases for adults") because the One Health Tool doesn't break down the cost.
184 | Solid culture and DST 177 | Cat. II Patient Kit A1 179 | Cat. II Patient Kit A2 181 | Treatment: second-line drugs 182 | MDR Fuel

@sakshimohan I'm working on this now. Is it possible to add 3HP (Isoniazid/Rifapentine) to the consumables database? It's not on the existing spreadsheet - it's a new recommendation for preventive therapy and is being used in Malawi I think.

@tdm32 There are a few HIV consumables for which I wasn't able to extract costs from the sources that I was using (CMST Catalogue, EHP Tool, 2016 One Health Tool). I think we can use the Global Fund Pooled Purchasing Mechanism Reference Prices. Another possible source is this CHAI ARV Reference Price list but this is older (2016). Would you have any thoughts on what would be the appropriate regimen for the following three item codes -
2671 | First-line ART regimen: adult 2672 | First line ART regimen:  child 2673 | First line ART regimen: young child
199 | Nevirapine, tablet, 200 mg - only the cost of oral solution ("Nevirapine 10mg/mL oral suspension, 100mL bottle") is available
Similarly, for TB I'm using the Stop TB partnership Catalogue (info on patient kits here). Cat I and III Patient Kits (item codes 176 and 178) are listed here so those were straightforward to cost. But the following TB consumables aren't listed. Could you advise on these? Note that for now I have included the cost as listed in the 2016 One Health Tool (but I had to use the unit 1 full treatment ("First line treatment for retreatment TB cases for adults") because the One Health Tool doesn't break down the cost.
184 | Solid culture and DST 177 | Cat. II Patient Kit A1 179 | Cat. II Patient Kit A2 181 | Treatment: second-line drugs 182 | MDR Fuel

@sakshimohan I'm working on this now. Is it possible to add 3HP (Isoniazid/Rifapentine) to the consumables database? It's not on the existing spreadsheet - it's a new recommendation for preventive therapy and is being used in Malawi I think.

branch consumables/editing_units_required_hiv
now has the updated item quantities for HIV and TB. Please note TB now requires new consumable 3HP.

@tdm32 There are a few HIV consumables for which I wasn't able to extract costs from the sources that I was using (CMST Catalogue, EHP Tool, 2016 One Health Tool). I think we can use the Global Fund Pooled Purchasing Mechanism Reference Prices. Another possible source is this CHAI ARV Reference Price list but this is older (2016). Would you have any thoughts on what would be the appropriate regimen for the following three item codes -
2671 | First-line ART regimen: adult 2672 | First line ART regimen:  child 2673 | First line ART regimen: young child
199 | Nevirapine, tablet, 200 mg - only the cost of oral solution ("Nevirapine 10mg/mL oral suspension, 100mL bottle") is available
Similarly, for TB I'm using the Stop TB partnership Catalogue (info on patient kits here). Cat I and III Patient Kits (item codes 176 and 178) are listed here so those were straightforward to cost. But the following TB consumables aren't listed. Could you advise on these? Note that for now I have included the cost as listed in the 2016 One Health Tool (but I had to use the unit 1 full treatment ("First line treatment for retreatment TB cases for adults") because the One Health Tool doesn't break down the cost.
184 | Solid culture and DST 177 | Cat. II Patient Kit A1 179 | Cat. II Patient Kit A2 181 | Treatment: second-line drugs 182 | MDR Fuel

I think it's good to stick with the full kits rather than breaking them down to individual tablets. The kits are supplied as full treatment courses. I was able to match with your costing data - let me know if it looks ok.

assert that the keys of a dict are the same as the columns of a dataframe

@tdm32 There are a few HIV consumables for which I wasn't able to extract costs from the sources that I was using (CMST Catalogue, EHP Tool, 2016 One Health Tool). I think we can use the Global Fund Pooled Purchasing Mechanism Reference Prices. Another possible source is this CHAI ARV Reference Price list but this is older (2016). Would you have any thoughts on what would be the appropriate regimen for the following three item codes -
2671 | First-line ART regimen: adult 2672 | First line ART regimen:  child 2673 | First line ART regimen: young child
199 | Nevirapine, tablet, 200 mg - only the cost of oral solution ("Nevirapine 10mg/mL oral suspension, 100mL bottle") is available
Similarly, for TB I'm using the Stop TB partnership Catalogue (info on patient kits here). Cat I and III Patient Kits (item codes 176 and 178) are listed here so those were straightforward to cost. But the following TB consumables aren't listed. Could you advise on these? Note that for now I have included the cost as listed in the 2016 One Health Tool (but I had to use the unit 1 full treatment ("First line treatment for retreatment TB cases for adults") because the One Health Tool doesn't break down the cost.
184 | Solid culture and DST 177 | Cat. II Patient Kit A1 179 | Cat. II Patient Kit A2 181 | Treatment: second-line drugs 182 | MDR Fuel

@sakshimohan I'm working on this now. Is it possible to add 3HP (Isoniazid/Rifapentine) to the consumables database? It's not on the existing spreadsheet - it's a new recommendation for preventive therapy and is being used in Malawi I think.

branch consumables/editing_units_required_hiv now has the updated item quantities for HIV and TB. Please note TB now requires new consumable 3HP.

Hi @tdm32. Thanks for updating the HIV consumables. I have now added the following ARV costs -

Item_code
Consumable_name_tlo
Cost (USD)
Units

2671
First-line ART regimen: adult
0.667
1 tablet   (Abacavir/Dolutegravir/Lamivudine 600/50/300mg)

2672
First line ART   regimen: older child
0.083
1 tablet   (Abacavir/Dolutegravir/Lamivudine 60/5/30mg)

2673
First line ART   regimen: young child
0.083
1 tablet   (Abacavir/Dolutegravir/Lamivudine 60/5/30mg)

For cotrimoxazole, can I confirm if you have expressed the dosage in terms of number of mg needed? I am not sure if I'm reading it correctly but it seems like you might have used tablets. Because I couldn't find the cost for this drug in either Global Fund Pooled Purchasing Mechanism Reference Prices or CHAI ARV Reference Price list, I have used CMST price for "Cotrimoxazole 480mg" and expressed cost for "1 mg".

Item_code
Consumable_name_tlo
Cost (USD)
Units

203
Cotrimoxazole 120mg_1000_CMST
0.001
1 mg

@tdm32 There are a few HIV consumables for which I wasn't able to extract costs from the sources that I was using (CMST Catalogue, EHP Tool, 2016 One Health Tool). I think we can use the Global Fund Pooled Purchasing Mechanism Reference Prices. Another possible source is this CHAI ARV Reference Price list but this is older (2016). Would you have any thoughts on what would be the appropriate regimen for the following three item codes -
2671 | First-line ART regimen: adult 2672 | First line ART regimen:  child 2673 | First line ART regimen: young child
199 | Nevirapine, tablet, 200 mg - only the cost of oral solution ("Nevirapine 10mg/mL oral suspension, 100mL bottle") is available
Similarly, for TB I'm using the Stop TB partnership Catalogue (info on patient kits here). Cat I and III Patient Kits (item codes 176 and 178) are listed here so those were straightforward to cost. But the following TB consumables aren't listed. Could you advise on these? Note that for now I have included the cost as listed in the 2016 One Health Tool (but I had to use the unit 1 full treatment ("First line treatment for retreatment TB cases for adults") because the One Health Tool doesn't break down the cost.
184 | Solid culture and DST 177 | Cat. II Patient Kit A1 179 | Cat. II Patient Kit A2 181 | Treatment: second-line drugs 182 | MDR Fuel

I think it's good to stick with the full kits rather than breaking them down to individual tablets. The kits are supplied as full treatment courses. I was able to match with your costing data - let me know if it looks ok.

TB looks good. Happy to stick with full kits. I feel confident about the costs of the following drugs which are extracted from Stop TB partnership Catalogue

178
Cat. I &   III Patient Kit B

176
Cat. I & III   Patient Kit A

182
MDR Fuel

However, for these ones, we've relied upon the 2016 One Health tool estimates for full treatment. I think it's fine to stick with this for now given that we don't have a better data source. But just mentioning it here for something to be mindful of.

177
Cat. II Patient   Kit A1

179
Cat. II Patient Kit   A2

181
Treatment:   second-line drugs

For 3HP (Isoniazid/Rifapentine), I haven't found cost or availability data yet, but I have added it to the ResourceFile_Consumables_Items_and_Packages.csv with item_code 2675 (c666e50). We might have to make assumptions if we can't find data. I will come back to this after Easter :)

@sakshimohan (@tbhallett @EvaJanouskova @andrew-phillips-1) - just realised that consumables for the cancer modules are not currently in master but are in #1124 (so aren't in this branch). Wondering if we should pull this out as a separate PR to go into master??

@tdm32 There are a few HIV consumables for which I wasn't able to extract costs from the sources that I was using (CMST Catalogue, EHP Tool, 2016 One Health Tool). I think we can use the Global Fund Pooled Purchasing Mechanism Reference Prices. Another possible source is this CHAI ARV Reference Price list but this is older (2016). Would you have any thoughts on what would be the appropriate regimen for the following three item codes -
2671 | First-line ART regimen: adult 2672 | First line ART regimen:  child 2673 | First line ART regimen: young child
199 | Nevirapine, tablet, 200 mg - only the cost of oral solution ("Nevirapine 10mg/mL oral suspension, 100mL bottle") is available
Similarly, for TB I'm using the Stop TB partnership Catalogue (info on patient kits here). Cat I and III Patient Kits (item codes 176 and 178) are listed here so those were straightforward to cost. But the following TB consumables aren't listed. Could you advise on these? Note that for now I have included the cost as listed in the 2016 One Health Tool (but I had to use the unit 1 full treatment ("First line treatment for retreatment TB cases for adults") because the One Health Tool doesn't break down the cost.
184 | Solid culture and DST 177 | Cat. II Patient Kit A1 179 | Cat. II Patient Kit A2 181 | Treatment: second-line drugs 182 | MDR Fuel

@sakshimohan I'm working on this now. Is it possible to add 3HP (Isoniazid/Rifapentine) to the consumables database? It's not on the existing spreadsheet - it's a new recommendation for preventive therapy and is being used in Malawi I think.

branch consumables/editing_units_required_hiv now has the updated item quantities for HIV and TB. Please note TB now requires new consumable 3HP.

Hi @tdm32. Thanks for updating the HIV consumables. I have now added the following ARV costs -

Item_code	Consumable_name_tlo	Cost (USD)	Units
2671	First-line ART regimen: adult	0.667	1 tablet (Abacavir/Dolutegravir/Lamivudine 600/50/300mg)
2672	First line ART regimen: older child	0.083	1 tablet (Abacavir/Dolutegravir/Lamivudine 60/5/30mg)
2673	First line ART regimen: young child	0.083	1 tablet (Abacavir/Dolutegravir/Lamivudine 60/5/30mg)
For cotrimoxazole, can I confirm if you have expressed the dosage in terms of number of mg needed? I am not sure if I'm reading it correctly but it seems like you might have used tablets. Because I couldn't find the cost for this drug in either Global Fund Pooled Purchasing Mechanism Reference Prices or CHAI ARV Reference Price list, I have used CMST price for "Cotrimoxazole 480mg" and expressed cost for "1 mg".

Item_code	Consumable_name_tlo	Cost (USD)	Units
203	Cotrimoxazole 120mg_1000_CMST	0.001	1 mg

I've changed the units now to use mg and not tablets

@tdm32 There are a few HIV consumables for which I wasn't able to extract costs from the sources that I was using (CMST Catalogue, EHP Tool, 2016 One Health Tool). I think we can use the Global Fund Pooled Purchasing Mechanism Reference Prices. Another possible source is this CHAI ARV Reference Price list but this is older (2016). Would you have any thoughts on what would be the appropriate regimen for the following three item codes -
2671 | First-line ART regimen: adult 2672 | First line ART regimen:  child 2673 | First line ART regimen: young child
199 | Nevirapine, tablet, 200 mg - only the cost of oral solution ("Nevirapine 10mg/mL oral suspension, 100mL bottle") is available
Similarly, for TB I'm using the Stop TB partnership Catalogue (info on patient kits here). Cat I and III Patient Kits (item codes 176 and 178) are listed here so those were straightforward to cost. But the following TB consumables aren't listed. Could you advise on these? Note that for now I have included the cost as listed in the 2016 One Health Tool (but I had to use the unit 1 full treatment ("First line treatment for retreatment TB cases for adults") because the One Health Tool doesn't break down the cost.
184 | Solid culture and DST 177 | Cat. II Patient Kit A1 179 | Cat. II Patient Kit A2 181 | Treatment: second-line drugs 182 | MDR Fuel

I think it's good to stick with the full kits rather than breaking them down to individual tablets. The kits are supplied as full treatment courses. I was able to match with your costing data - let me know if it looks ok.

TB looks good. Happy to stick with full kits. I feel confident about the costs of the following drugs which are extracted from Stop TB partnership Catalogue

178	Cat. I & III Patient Kit B
176	Cat. I & III Patient Kit A
182	MDR Fuel
However, for these ones, we've relied upon the 2016 One Health tool estimates for full treatment. I think it's fine to stick with this for now given that we don't have a better data source. But just mentioning it here for something to be mindful of.

177	Cat. II Patient Kit A1
179	Cat. II Patient Kit A2
181	Treatment: second-line drugs
For 3HP (Isoniazid/Rifapentine), I haven't found cost or availability data yet, but I have added it to the ResourceFile_Consumables_Items_and_Packages.csv with item_code 2678 (c666e50). We might have to make assumptions if we can't find data. I will come back to this after Easter :)
Update (3 April 2024) Hi @tdm32 I have now included the cost of 3HP from this publication (Frick (2020) - An activist's guide to Rifapentine for the treatment of TB infection). I've used the Global Fund ERP endorsed fixed dose combination (Isoniazid 300mg/ Rifapentine 300mg) which costs $15 for a blister of 36 tablets (one patient course). The cost is expressed per single tablet (i.e. $0.42) rather than full course because I assume that the dosage would depend on age and weight? Let me know if you prefer the cost to be expressed as on full treatment.
For availability, I assumed that the availability of "Isoniazid 300mg/ Rifapentine 300mg" is the same as the availability of "Isoniazid tablets, 300mg" OR "Isoniazid tablets, 100mg" as per the OpenLMIS data. Because data on Rifapentine is not available from the 2018 datasets which we have. Here are the average availability estimates based on LMIS - Facility_level_1a | 0.581922 Facility_level_1b | 0.576664 Facility_level_2 | 0.841197 Facility_level_3 | 0.800000
This is slightly lower than the availability estimates we had for 'Isoniazid/Pyridoxine, tablet 300 mg' because this was based on the availability of 'Isoniazid' and 'Pyridoxine' Facility_level_1a | 0.402094 Facility_level_1b | 0.547532 Facility_level_2 | 0.718991 Facility_level_3 | 0.767792

Great - I've added this now and happy to use the tablet as the single unit. The dosage is 1 tablet per week for all.

@sakshimohan branch consumables/editing_units_required_hiv now ready for your review. It contains the updated units for HIV and TB.

@sakshimohan (@tbhallett @EvaJanouskova @andrew-phillips-1) - just realised that consumables for the cancer modules are not currently in master but are in #1124 (so aren't in this branch). Wondering if we should pull this out as a separate PR to go into master??

Hi @joehcollins. Happy to take your advice on this. But I think that if it's just these three consumables to add (endoscope, prostate specific antigen test, and cystoscope), I would add cost data to this PR to keep all consumable costing edits on one branch.

Sorry @sakshimohan it wasnt about adding the consumables to the consumable spreadsheet - it was that the requests for all releveant consumables for the cancer modules are not currently in master. They were added in the above PR. So we just need to think about maybe adding them in here? unless the equipment PR is going to go in before this which i'm not sure on...

@sakshimohan @tbhallett @tdm32 - just wanted to flag here that I think its going to be very difficult to meaningfully estimate 'dose' or 'units' of oxygen. Clinically O2 is delivered in litres/min and the 'prescribed' l/min varies significantly depending on indication (with highest dose being 15/L for non-ventiltated patients). So we'd need to estimate the required O2 'dose' along with the potential number of minutes an individual would require O2 for. We might need to just do an apporixamtion saying all patients get 8L/min for 2 days (2880mins * 8 = 23040)?

@sakshimohan @tbhallett @tdm32 - just wanted to flag here that I think its going to be very difficult to meaningfully estimate 'dose' or 'units' of oxygen. Clinically O2 is delivered in litres/min and the 'prescribed' l/min varies significantly depending on indication (with highest dose being 15/L for non-ventiltated patients). So we'd need to estimate the required O2 'dose' along with the potential number of minutes an individual would require O2 for. We might need to just do an apporixamtion saying all patients get 8L/min for 2 days (2880mins * 8 = 23040)?

@joehcollins I think it's fine to approximate use - I'm doing this with other consumables too. There's also wastage that we don't factor in, so we will never be super precise on these estimates.

@sakshimohan for the safety boxes required for used syringes, the unit is 1. This will suffice for approx 100 syringes. It it possible to change the single unit to 0.01 of a safety box, or would this violate the logic that you have in place?

@sakshimohan @tbhallett @tdm32 - just wanted to flag here that I think its going to be very difficult to meaningfully estimate 'dose' or 'units' of oxygen. Clinically O2 is delivered in litres/min and the 'prescribed' l/min varies significantly depending on indication (with highest dose being 15/L for non-ventiltated patients). So we'd need to estimate the required O2 'dose' along with the potential number of minutes an individual would require O2 for. We might need to just do an apporixamtion saying all patients get 8L/min for 2 days (2880mins * 8 = 23040)?

I suggest also running this past Ines and Tim C

Thanks Andrew - will do this for ALRI but theres also O2 in my modules and i'm not so sure on that either! I will estimate as above for now

@sakshimohan branch consumables/editing_units_required_schisto_epi_measles now has the updated units for these three conditions.

Thanks @joehcollins for raising that up. Indeed that block of code was one my to-do's in that module. You may proceed as suggested. Thanks

@sakshimohan branch consumables/editing_units_required_hiv now ready for your review. It contains the updated units for HIV and TB.

Thank you, @tdm32. All looks great! I've merged the consumables/editing_units_required_hiv into this PR.

@sakshimohan for the safety boxes required for used syringes, the unit is 1. This will suffice for approx 100 syringes. It it possible to change the single unit to 0.01 of a safety box, or would this violate the logic that you have in place?

That's absolutely fine, @tdm32. I've now updated the costing RF to express the cost of item code 154 in terms of "1 disposed syringe (100 syringes per box)".

Item_Code
Consumable_name_tlo
Chosen_unit_for_price
Final_price_per_chosen_unit   (USD, 2023)
Data_source_for_final_price

31
Methylergometrine,   Injection 0.2 mg/ml, 1 ml amp
1 ml ampoule
0.025
One Health Tool 2016

32
Pethidine, 50 mg/ml,   2 ml ampoule
1 ampoule of 2ml (50 mg/ml)
0.328
CMST 2023

36
Epinephrine, ampoule,   1 mg/ml
1 ampoule of 4ml (1 mg/ml)
0.176
CMST 2023

45
Albendazole   200mg_1000_CMST
1 mg
0.000
CMST 2023

131
Cephalotin, 100 ml   vial
1 vial of 1g
0.741
CMST 2023

132
Ampicillin injection   250mg, PFR_1_CMST
1 vial of 500mg
0.123
CMST 2023

160
Insecticide-treated   net
1 net
0.9517501
Malawi EHP 2021

@sakshimohan - consumable 294 (Beclomethasone diproprionate inhalation, 50mcg/dose, 200 doses) is requested by COPD but isnt in the costing spreadsheet. Possible to add? cheers

@sakshimohan - consumable 294 (Beclomethasone diproprionate inhalation, 50mcg/dose, 200 doses) is requested by COPD but isnt in the costing spreadsheet. Possible to add? cheers

ignore me - this consumable was actually never called in the module. i've now removed it.

If you decide to remove the Intervention_Pkg at the end, you will need to check where the fnc get_item_codes_from_package_name is used and changed that.

Hi @EvaJanouskova. We decided that for now we'll just keep the current structure for how we deal with consumables, i.e. specifying quantity needed within modules, and consider a more serious overhaul where everything is moved with RFs in the future. @tbhallett will raise a separate issue for me to update the ResourceFile_Consumables_Items_and_Packages.csv to remove Expected_units_per_case and Unit_Cost columns. Just to clarify, the plan is to retain the Intervention_Pkg column. (Issue #1303) We've said that it's lower in terms of priority for now so I'll reach out to you in due course so see if you want to retain any information in the columns you had manually added to ResourceFile_Consumables_Items_and_Packages.csv for your contraception work.

@sakshimohan, I'm not sure if it is worth to retain Intervention_Pkg column if Expected_units_per_case are removed. The fnc get_item_codes_from_package_name uses the the package name to get all item codes along with the units per case.
@tbhallett, If either is removed, Intervention_Pkg or Expected_units_per_case, we need to get rid of the fnc 'consumable_sets_for_pkgs' (in consumables.py), or we may re-code the fnc to lookup the set of items & the units per case in another RF by pkg name, smt like ResourceFile_Consumables_SetsForPkgs.csv which should be assured to contain only items from the ResourceFile_Consumables_Items_and_Packages.csv and the units per case in SetsForPkgs be related to the units from Items_and_Packages. What do you think?

@tbhallett @sakshimohan (@timcolbourn @andrew-phillips-1) - i've finished all the modules assigned to me as first or second module lead apart from ALRI.
I'm just chatting to Ines about to get her source for age-based-dosing but hopefully shouldnt be too hard to update as its currently structured.
Not sure how to sort out the conflict from COPD if anyone has any ideas?

I am also wondering if we want to have some kind of central repository which details each of the modelled interventions, the drugs being used, the assumed dose and the source. I know people will likely have things like this in their right up but it might be worth formalising this - especially as i've noticed there is some discrepancy with what is in the some modules and whats recommended in the most recent STG (2023)

@sakshimohan, I started with Contraception, please see some comments/questions below:
Contraception

I'm suggesting following changes in ResourceFile_Costing/consumables sheet:
- for item 2019 (pregnancy test) Intervention_Pkg to be 'Contraception initiation', and Intervention_Cat to be ‘Maternal/Newborn and Reproductive Health’.
- for item 0 (progestogen-only pills Levonorgestrel 0.0375 mg) Intervention_Pkg to be ‘Pills’, same as it is for the other pills there and same as it is for both pills in RF_Consumables_Items_and_Pkgs,.

@tbhallett Earlier we decided to join the two types of pills (items 0 and 1) to be one item to code it as any of them prescribed, and the price used as the weighted price of these two items, assuming that item 1 is used in 80% cases and item 0 in 20% cases. Do we want to apply it here as well, ie join these two items, or do we want to create a new joined item based on these two so no information is lost in the process, or should I rather code within the contraception module to be chosen which of the pills are provided?

I can see that the ResourceFile_Consumables_Items_and_Packages still contains both Intervention_Pkg and Expected_Units_Per_Case, does it mean I can keep using the fnc get_item_codes_from_package_name and deal with that in the separate PR where you want to remove the Expected_Units_Per_Case, is that so? (If so, I will suggest updates for Expected_Units_Per_Case in RF_..Items_and_Pkgs we now use with a reference to the Chosen_unit_for_price.) Or if already decided that the fnc get_item_codes_from_package_name won't be used at all in the future, I’ll recode it using the fnc get_item_code_from_item_name.

When will be the unit costs in RF_..Items_and_Pkgs matched to Final_price_per_chosen_unit from ResourceFile_Costing/consumables?

What units do the Expected_Units_Per_Case_Malawi_HBP refer to, is it recorded somewhere, or do they refer to Chosen_unit_for_price?

When I was figuring out the prices with Emi, she suggested some changes in EHP 2021 data spreadsheet, do you want them in case you would like to use them anyhow?

How about if an item is used for more than one Intervention_Pkg. For example, the Intervention_Pkg for item 98 (Water for injection) is now ‘Cesearian section with indication’, but we want this item to use for Intervention_Pkg ‘Injectable’ as well, and there are more such (item 75).

I’ve also noticed that you don’t have the price from EHP 2021 for items 5, and 98 though these items were in the spreadsheet, not sure if it is in purpose or not.

Is the function used for Final price per chosen unit correct? I think the price should be in MWK, but the first price looked upped, J3, is price in USD.

The chosen price for item 75, seems too high for 1 swab. And if you look up the price according to EHP or OHT it’s too different, I think it should have been divided by 100.

@sakshimohan, Do we know how many pills are in the monthly packet for each type of pills, combined (item 1) and progesterone only pills (item 0), it can be the 21-day or 28-day packet of pills with 7 inactive pills in the 28-day packet. So if we want the price by pill, I need to know the numbers of pills in the packets, or we have to change the unit to a monthly packet.

Noting here that we'll resolve the 'to-do's in cancer_consuambles.py in this PR
https://github.com/UCL/TLOmodel/pull/1353#issuecomment-2115426043

[ ] resolve to-do in cancer_consuambles.py

@sakshimohan, @tbhallett, The item names in ResourceFile_Costing/consumables and RF_Items&Pkgs sometimes differ. I’m using the names from RF_Items&Pkgs, is that correct (That’s where the function get_item_code_from_item_name looks up for the code after all.)?
For instance:

There is the original version of name of item 1 used in RF_Items&Pkgs, but an old version name of joined pill item (joining item 0 and 1) (updated version in master) is used in RF_Costing, while original item 0 is preserved as well in RF_Costing. This should be resolved, I’ll now use the version from RF_Items&Pkgs within the contraception module unless you tell me otherwise.
Item name for Female condom slightly differs in these two RFs, again I'm using name from the RF_Items&Pkgs unless you tell me otherwise.

@tbhallett @sakshimohan - my modules all done now. @tbhallett please check you're happy with the simplification in ALRI! thanks

@sakshimohan, @tbhallett Is it that we need to use only items from ResourceFile_Costing/consumables, right?
So if I used the item 1839 'Medroxyprogesterone acetate injection 150mg/mL, 1mL vial with 2ml syringe with 22g 0.7 X 25mm needle_each_CMST' from RF_Items&Pkgs, I need to replace it with the alternative item 3 'Depot-Medroxyprogesterone Acetate 150 mg - 3 monthly', is it? Does the costing for this item include all other stuff which were within the item I used originally, ie syringe and needle, or do I need to choose a syringe and a needle from other items in RF_Costing to be used along with the item 3?

@tbhallett @sakshimohan - my modules all done now. @tbhallett please check you're happy with the simplification in ALRI! thanks

Thanks @joehcollins  -- very happy with the simplifications in ALRI

@sakshimohan, @tbhallett Is it that we need to use only items from ResourceFile_Costing/consumables, right?
So if I used the item 1839 'Medroxyprogesterone acetate injection 150mg/mL, 1mL vial with 2ml syringe with 22g 0.7 X 25mm needle_each_CMST' from RF_Items&Pkgs, I need to replace it with the alternative item 3 'Depot-Medroxyprogesterone Acetate 150 mg - 3 monthly', is it? Does the costing for this item include all other stuff which were within the item I used originally, ie syringe and needle, or do I need to choose a syringe and a needle from other items in RF_Costing to be used along with the item 3?

I'll defer to @sakshimohan on the difference between the different lists. But, my perspective is that you're "allowed" to use any item_codes for Consumables that are defined in resources/healthsystem/consumables/ResourceFile_Consumables_Items_and_Packages.csv -- but the original version, without manual edits.

@sakshimohan,

The chosen unit for implant Jadelle (item 12) is '1 implant', do you mean by that both rods? 2 rods are inserted with this implant so I would call that 1 implant, if so, could you maybe specify that in the brackets so it is clear (eg chosen unit: ‘1 implant (2 rods)’)?

Can I use less then 1 unit if it doesn’t mean less than 1 unit but only use in some cases. For instance, if 1 unit used in half cases, is that okay if I use 0.5 unit of the item?

@sakshimohan, Please see the PR #1362.

Stunting: nothing need to be changed
Wasting: the version in master doesn't include any consumables

Noting here that we'll resolve the 'to-do's in cancer_consuambles.py in this PR #1353 (comment)

[ ]  resolve to-do in cancer_consuambles.py

@sakshimohan, Could you please replicate the changes you did before in the #1341 (commits 4d8712f & ff4d072), then I could proceed to resolve some of the TODOs. Thank you!

I am also wondering if we want to have some kind of central repository which details each of the modelled interventions, the drugs being used, the assumed dose and the source. I know people will likely have things like this in their right up but it might be worth formalising this - especially as i've noticed there is some discrepancy with what is in the some modules and whats recommended in the most recent STG (2023)

Thanks, @joehcollins. That's a great suggestion. I've created issue #1364 to make this update. Do you think the best way to do this would be to extract these values manually from the module script?

Hi @EvaJanouskova Many thanks for your detailed review and I'm sorry for the delay in getting back to you. I'm just getting back to this PR today. Please see my responses in-line below.

I'm suggesting following changes in ResourceFile_Costing/consumables sheet:

for item 2019 (pregnancy test) Intervention_Pkg to be 'Contraception initiation', and Intervention_Cat to be ‘Maternal/Newborn and Reproductive Health’.
for item 0 (progestogen-only pills Levonorgestrel 0.0375 mg) Intervention_Pkg to be ‘Pills’, same as it is for the other pills there and same as it is for both pills in RF_Consumables_Items_and_Pkgs,.

How about if an item is used for more than one Intervention_Pkg. For example, the Intervention_Pkg for item 98 (Water for injection) is now ‘Cesearian section with indication’, but we want this item to use for Intervention_Pkg ‘Injectable’ as well, and there are more such (item 75).

The Intervention_Cat and Intervention_Pkg columns are not used. They were simply extracted from the original consumables resource file to guide Newton and me with finding cost data from appropriate sources . There are consumables on the list which are used by more than one Intervention_Cat but they are listed only once in the costing spreadsheet. I plan to eventually delete these columns from ResourceFile_costing.xlsx.

I can see that the ResourceFile_Consumables_Items_and_Packages still contains both Intervention_Pkg and Expected_Units_Per_Case, does it mean I can keep using the fnc get_item_codes_from_package_name and deal with that in the separate PR where you want to remove the Expected_Units_Per_Case, is that so? (If so, I will suggest updates for Expected_Units_Per_Case in RF_..Items_and_Pkgs we now use with a reference to the Chosen_unit_for_price.) Or if already decided that the fnc get_item_codes_from_package_name won't be used at all in the future, I’ll recode it using the fnc get_item_code_from_item_name.
edited: Because of the mix up with manual changes in RF_Items&Pkgs, I'm rather going to re-code it using the item names instead of pkg name, at least for now.

What units do the Expected_Units_Per_Case_Malawi_HBP refer to, is it recorded somewhere, or do they refer to Chosen_unit_for_price?

Again the Expected_Units_Per_Case column is redundant and will be deleted. Initially we had planned to put together data on Expected Units per case in the costing spreadsheet but then we reverted to our current plan of including consumable quantities required within module scripts.
These values were extracted from Emi's 2021 EHP costing. Note that we only extracted data from Emi's spreadsheet where cost data was not available from CMST, which was source primary source of cost data. Data_source_for_final_price indicates where the final cost data is coming from.

When will be the unit costs in RF_..Items_and_Pkgs matched to Final_price_per_chosen_unit from ResourceFile_Costing/consumables?

The unit costs in RF_..Items_and_Pkgs will not be used. And only ResourceFile_Costing will be used by the costing module.

When I was figuring out the prices with Emi, she suggested some changes in EHP 2021 data spreadsheet, do you want them in case you would like to use them anyhow?

Yes, please feel free to forward me your communication with her and I can see if anything needs to be updated. As I said, we only relied on her costing data where we didn't find new cost information from CMST's 2023 catalogue so these changes wil only apply where we have used her data.

I’ve also noticed that you don’t have the price from EHP 2021 for items 5, and 98 though these items were in the spreadsheet, not sure if it is in purpose or not.

This is because we already had data from CMST.

Is the function used for Final price per chosen unit correct? I think the price should be in MWK, but the first price looked upped, J3, is price in USD.

The conversion rate is applied to only values taken from the CMST and Emi's EHP data which are expressed in MWK (so if something comes from column J, only the inflation rate is applied to it). For example, see item 1220.

The chosen price for item 75, seems too high for 1 swab. And if you look up the price according to EHP or OHT it’s too different, I think it should have been divided by 100.

Thanks, that's a good spot! I'll divide that by 100.

Some item names feel to be a bit confusing, as containing the unit which is not the chosen unit, e.g. item 5 'Povidone iodine, solution, 10 %, 5 ml per injection', a bit confusing name as the chosen unit is 1 ml. Couldn't be either the amounts in names replaced by the chosen unit, or used as the chosen unit?

I'm reluctant to change the original names of consumables because this will require further changes to be made to modules which call consumables using their names rather than item code.

@sakshimohan,

Do we know how many pills are in the monthly packet for each type of pills, combined (item 1) and progesterone only pills (item 0), it can be the 21-day or 28-day packet of pills with 7 inactive pills in the 28-day packet. So if we want the price by pill, I need to know the numbers of pills in the packets, or we have to change the unit to a monthly packet.
(The prices we had from Emi don't fit with your final price (corresponding to 9 pills and 7 pills, respectively) so I can't derive the numbers from that.)
Could we find out whether the trocar (item 10) for which we have the cost is reusable or not? (We used the amount 0.1 which indicates 10 uses on average. ) If so, we need to either change the chosen unit to 1 use, or move from consumables to equipment.

The prices for item codes 0 and 1 come from CMST. The consumable names and prices in the CMST Price catalogue are as follows -
0 - Microlut (Levonorgestrel 0.03mg) - MWK 49.91
1 - Ethinylestradiol 0.03mg + levonorgestrel 0.15mg - MWK 59.05
My understanding is that the price is for each individual pill because the CMST's unit specification is "EACH". So in your module you'd just need to specify the number of pills needed per patient. But let me know if you think it's different.
Let me discuss the Trocar question with Tim on my call with him today.

RF_Items&Pkgs

Hi @EvaJanouskova. As @tbhallett said, resources/healthsystem/consumables/ResourceFile_Consumables_Items_and_Packages.csv remains the go to reference for names of consumables and item codes. The only columns relevant in ResourceFile_Costing.xlsx/consumables are Item_Code and Final_price_per_chosen_unit (USD, 2023). You should stick with the consumable from ResourceFile_Consumables_Items_and_Packages.csv which is most relevant to your module. If you don't find this consumable in ResourceFile_Costing.xlsx/consumables, let me know here and I will add it in.
Also see this request  from Tara which was addressed through this commit. So if while going through your consumables you find something that is neither in ResourceFile_Consumables_Items_and_Packages.csv nor in ResourceFile_Costing.xlsx/consumables, I can still add that in - it would just require a few extra changes.

ResourceFile_Consumables_Items_and_Packages.csv

As mentioned above, you're right in sticking with the name in ResourceFile_Consumables_Items_and_Packages.csv

@sakshimohan,

Do we know how many pills are in the monthly packet for each type of pills, combined (item 1) and progesterone only pills (item 0), it can be the 21-day or 28-day packet of pills with 7 inactive pills in the 28-day packet. So if we want the price by pill, I need to know the numbers of pills in the packets, or we have to change the unit to a monthly packet.
(The prices we had from Emi don't fit with your final price (corresponding to 9 pills and 7 pills, respectively) so I can't derive the numbers from that.)

The prices for item codes 0 and 1 come from CMST. The consumable names and prices in the CMST Price catalogue are as follows - 0 - Microlut (Levonorgestrel 0.03mg) - MWK 49.91 1 - Ethinylestradiol 0.03mg + levonorgestrel 0.15mg - MWK 59.05 My understanding is that the price is for each individual pill because the CMST's unit specification is "EACH". So in your module you'd just need to specify the number of pills needed per patient. But let me know if you think it's different.

Exactly, but to specify the number of pills, I need to know which packets are they distributing in Malawi, if those are 21-day or 28-day packets. Can we find that information?

@sakshimohan,

Do we know how many pills are in the monthly packet for each type of pills, combined (item 1) and progesterone only pills (item 0), it can be the 21-day or 28-day packet of pills with 7 inactive pills in the 28-day packet. So if we want the price by pill, I need to know the numbers of pills in the packets, or we have to change the unit to a monthly packet.
(The prices we had from Emi don't fit with your final price (corresponding to 9 pills and 7 pills, respectively) so I can't derive the numbers from that.)

The prices for item codes 0 and 1 come from CMST. The consumable names and prices in the CMST Price catalogue are as follows - 0 - Microlut (Levonorgestrel 0.03mg) - MWK 49.91 1 - Ethinylestradiol 0.03mg + levonorgestrel 0.15mg - MWK 59.05 My understanding is that the price is for each individual pill because the CMST's unit specification is "EACH". So in your module you'd just need to specify the number of pills needed per patient. But let me know if you think it's different.

Exactly, but to specify the number of pills, I need to know which packets are they distributing in Malawi, if those are 21-day or 28-day packets. Can we find that information?

Ah sorry for misunderstanding your question. Your best bet would be to ask Emi. I've looked at the 2015 Malawi Standard treatment guidelines and coudn't find this ((Levonorgestrel is listed as an emergency contraceptive only on page 292)). You could also look for international guidelines to specify quantity.

@sakshimohan,

The chosen unit for implant Jadelle (item 12) is '1 implant', do you mean by that both rods? 2 rods are inserted with this implant so I would call that 1 implant, if so, could you maybe specify that in the brackets so it is clear (eg chosen unit: ‘1 implant (2 rods)’)?

Thanks, Eva. I'll make that update so that the cost is expressed in terms of 1 implant (2 rods). Note that I will make this change as well as the change for gauze swabs (divide the cost by 100) in the costing PR #1279 as that's where the final costing RF is maintained. I've update the units in the table above for reference.

Can I use less then 1 unit if it doesn’t mean less than 1 unit but only use in some cases. For instance, if 1 unit used in half cases, is that okay if I use 0.5 unit of the item?

Yes, that's absolutely fine.

Noting here that we'll resolve the 'to-do's in cancer_consuambles.py in this PR #1353 (comment)

[ ]   resolve to-do in cancer_consuambles.py

@sakshimohan, Could you please replicate the changes you did before in the #1341 (commits 4d8712f & ff4d072), then I could proceed to resolve some of the TODOs. Thank you!

@EvaJanouskova This is now done. Note that 4d8712f was already in because this is a change linked to an update in the dropbox file ResouceFile_hhfa_consumables.xlsx which is why you don't see a separate commit for this above.

I am also wondering if we want to have some kind of central repository which details each of the modelled interventions, the drugs being used, the assumed dose and the source. I know people will likely have things like this in their right up but it might be worth formalising this - especially as i've noticed there is some discrepancy with what is in the some modules and whats recommended in the most recent STG (2023)

Thanks, @joehcollins. That's a great suggestion. I've created issue #1364 to make this update. Do you think the best way to do this would be to extract these values manually from the module script?
Ignore this Joe. @tbhallett has suggested that Matt who is working on re-extracting resource info to population this page will be able to compile this information.

Ok perfect!!

@sakshimohan,

Do we know how many pills are in the monthly packet for each type of pills, combined (item 1) and progesterone only pills (item 0), it can be the 21-day or 28-day packet of pills with 7 inactive pills in the 28-day packet. So if we want the price by pill, I need to know the numbers of pills in the packets, or we have to change the unit to a monthly packet.
(The prices we had from Emi don't fit with your final price (corresponding to 9 pills and 7 pills, respectively) so I can't derive the numbers from that.)
Could we find out whether the trocar (item 10) for which we have the cost is reusable or not? (We used the amount 0.1 which indicates 10 uses on average. ) If so, we need to either change the chosen unit to 1 use, or move from consumables to equipment.

The prices for item codes 0 and 1 come from CMST. The consumable names and prices in the CMST Price catalogue are as follows - 0 - Microlut (Levonorgestrel 0.03mg) - MWK 49.91 1 - Ethinylestradiol 0.03mg + levonorgestrel 0.15mg - MWK 59.05 My understanding is that the price is for each individual pill because the CMST's unit specification is "EACH". So in your module you'd just need to specify the number of pills needed per patient. But let me know if you think it's different.
Let me discuss the Trocar question with Tim on my call with him today. The decision might be based on convinience here though. Because we currently have Trocar coded as a consumable in the availability data as well as costing, it might be easiest to stick with this. The costing will be correct as long as we include 0.1 as number of units required.

@sakshimohan, I've discussed this with Tim H and he thinks it's best if the unit is 1 use, and then you can adjust the price per 1 use depending on whether you're using a reusable or a disposable trocar for the cost calculation.

You should stick with the consumable from ResourceFile_Consumables_Items_and_Packages.csv which is most relevant to your module. If you don't find this consumable in ResourceFile_Costing.xlsx/consumables, let me know here and I will add it in.

consumables from ResourceFile_Consumables_Items_and_Packages.csv we use for cancer modules, but are not in  ResourceFile_Costing.xlsx/consumables (at least not in the table above):

Biopsy needle
Specimen container

@sakshimohan,

Do we know how many pills are in the monthly packet for each type of pills, combined (item 1) and progesterone only pills (item 0), it can be the 21-day or 28-day packet of pills with 7 inactive pills in the 28-day packet. So if we want the price by pill, I need to know the numbers of pills in the packets, or we have to change the unit to a monthly packet.
(The prices we had from Emi don't fit with your final price (corresponding to 9 pills and 7 pills, respectively) so I can't derive the numbers from that.)
Could we find out whether the trocar (item 10) for which we have the cost is reusable or not? (We used the amount 0.1 which indicates 10 uses on average. ) If so, we need to either change the chosen unit to 1 use, or move from consumables to equipment.

The prices for item codes 0 and 1 come from CMST. The consumable names and prices in the CMST Price catalogue are as follows - 0 - Microlut (Levonorgestrel 0.03mg) - MWK 49.91 1 - Ethinylestradiol 0.03mg + levonorgestrel 0.15mg - MWK 59.05 My understanding is that the price is for each individual pill because the CMST's unit specification is "EACH". So in your module you'd just need to specify the number of pills needed per patient. But let me know if you think it's different.
Let me discuss the Trocar question with Tim on my call with him today. The decision might be based on convinience here though. Because we currently have Trocar coded as a consumable in the availability data as well as costing, it might be easiest to stick with this. The costing will be correct as long as we include 0.1 as number of units required.

@sakshimohan, I've discussed this with Tim H and he thinks it's best if the unit is 1 use, and then you can adjust the price per 1 use depending on whether you're using a reusable or a disposable trocar for the cost calculation.

Thanks, @EvaJanouskova. I have updated the cost of the trocar assuming 100 uses per reuseable piece (reference) and a cost of  Euros 95  (reference) per piece in the costing PR #1279 (commit 5c007cc).  This gives a cost of approx. $1.27 per use. The new unit is 1 use of reuseable trocar (assuming 100 uses per trocar).
@joehcollins Do you use trocar in your modules? If yes, could you update your units in reference to this revision.

You should stick with the consumable from ResourceFile_Consumables_Items_and_Packages.csv which is most relevant to your module. If you don't find this consumable in ResourceFile_Costing.xlsx/consumables, let me know here and I will add it in.

consumables from ResourceFile_Consumables_Items_and_Packages.csv we use for cancer modules, but are not in ResourceFile_Costing.xlsx/consumables (at least not in the table above):

Biopsy needle
Specimen container
@EvaJanouskova
Added to PR https://github.com/UCL/TLOmodel/pull/1279 (commit 63e8e46)
Units - 1 needle (biopsy needle), 1 container (specimen container)

@sakshimohan  -- we think we're done with this.
@tbhallett will double-check calibrations.
@sakshimohan please check that these changes are the ones you want to come in on this PR.
Hooray!!!!

Master at current tip (#1273 just merged)
This branch

Master at current tip (#1273 just merged)
This branch

No evidence of wild changes in the calibrations being introduced in this branch ✅

@sakshimohan  -- this is OK with me to merge, if you're also happy?
(I presume you're happy with the change in the scripts/resource file that you manage).


Can't call openEditor from onselection events

Hi, I'm very new to jexcel and having trouble with openEditor.
I know that selecting a certain cell and, press down any key will be enough to input a value in the cell.
But just selecting a cell won't activate IME and can't input a Japanese character from first input.
I tried to figure this problem by using console log and jexcel events, and found that openEditor was called after entering a first character. Not by just clicking the cell.
So I thought using onselection event and call openEditor will solve this problem. But it didn't work.
I couldn't find the reason, but closeEditor was called from openEditor .
https://github.com/paulhodel/jexcel/blob/master/dist/jexcel.js#L1617
By double clicking the cell will open the editor, so I can enter the Japanese character from first input, but  is there a way to call openEditor by just clicking the cell?
or be able to input a japanese character from first character, instead of double clicking?

        <meta charset="utf8">

    <link rel="stylesheet" href="css/jexcel.css" type="text/css" />

    <link rel="stylesheet" href="css/jsuites.css" type="text/css" />

GIF
C1: click
C2: double click
C3: click

There is a method call
validLetter()
This method must return true to start the edition. You are invited to add the codes you think are missing here.
jexcel.validLetter = function(text) {
    var regex = /([\u0041-\u005A\u0061-\u007A\u00AA\u00B5\u00BA\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02C1\u02C6-\u02D1\u02E0-\u02E4\u02EC\u02EE\u0370-\u0374\u0376\u0377\u037A-\u037D\u0386\u0388-\u038A\u038C\u038E-\u03A1\u03A3-\u03F5\u03F7-\u0481\u048A-\u0527\u0531-\u0556\u0559\u0561-\u0587\u05D0-\u05EA\u05F0-\u05F2\u0620-\u064A\u066E\u066F\u0671-\u06D3\u06D5\u06E5\u06E6\u06EE\u06EF\u06FA-\u06FC\u06FF\u0710\u0712-\u072F\u074D-\u07A5\u07B1\u07CA-\u07EA\u07F4\u07F5\u07FA\u0800-\u0815\u081A\u0824\u0828\u0840-\u0858\u08A0\u08A2-\u08AC\u0904-\u0939\u093D\u0950\u0958-\u0961\u0971-\u0977\u0979-\u097F\u0985-\u098C\u098F\u0990\u0993-\u09A8\u09AA-\u09B0\u09B2\u09B6-\u09B9\u09BD\u09CE\u09DC\u09DD\u09DF-\u09E1\u09F0\u09F1\u0A05-\u0A0A\u0A0F\u0A10\u0A13-\u0A28\u0A2A-\u0A30\u0A32\u0A33\u0A35\u0A36\u0A38\u0A39\u0A59-\u0A5C\u0A5E\u0A72-\u0A74\u0A85-\u0A8D\u0A8F-\u0A91\u0A93-\u0AA8\u0AAA-\u0AB0\u0AB2\u0AB3\u0AB5-\u0AB9\u0ABD\u0AD0\u0AE0\u0AE1\u0B05-\u0B0C\u0B0F\u0B10\u0B13-\u0B28\u0B2A-\u0B30\u0B32\u0B33\u0B35-\u0B39\u0B3D\u0B5C\u0B5D\u0B5F-\u0B61\u0B71\u0B83\u0B85-\u0B8A\u0B8E-\u0B90\u0B92-\u0B95\u0B99\u0B9A\u0B9C\u0B9E\u0B9F\u0BA3\u0BA4\u0BA8-\u0BAA\u0BAE-\u0BB9\u0BD0\u0C05-\u0C0C\u0C0E-\u0C10\u0C12-\u0C28\u0C2A-\u0C33\u0C35-\u0C39\u0C3D\u0C58\u0C59\u0C60\u0C61\u0C85-\u0C8C\u0C8E-\u0C90\u0C92-\u0CA8\u0CAA-\u0CB3\u0CB5-\u0CB9\u0CBD\u0CDE\u0CE0\u0CE1\u0CF1\u0CF2\u0D05-\u0D0C\u0D0E-\u0D10\u0D12-\u0D3A\u0D3D\u0D4E\u0D60\u0D61\u0D7A-\u0D7F\u0D85-\u0D96\u0D9A-\u0DB1\u0DB3-\u0DBB\u0DBD\u0DC0-\u0DC6\u0E01-\u0E30\u0E32\u0E33\u0E40-\u0E46\u0E81\u0E82\u0E84\u0E87\u0E88\u0E8A\u0E8D\u0E94-\u0E97\u0E99-\u0E9F\u0EA1-\u0EA3\u0EA5\u0EA7\u0EAA\u0EAB\u0EAD-\u0EB0\u0EB2\u0EB3\u0EBD\u0EC0-\u0EC4\u0EC6\u0EDC-\u0EDF\u0F00\u0F40-\u0F47\u0F49-\u0F6C\u0F88-\u0F8C\u1000-\u102A\u103F\u1050-\u1055\u105A-\u105D\u1061\u1065\u1066\u106E-\u1070\u1075-\u1081\u108E\u10A0-\u10C5\u10C7\u10CD\u10D0-\u10FA\u10FC-\u1248\u124A-\u124D\u1250-\u1256\u1258\u125A-\u125D\u1260-\u1288\u128A-\u128D\u1290-\u12B0\u12B2-\u12B5\u12B8-\u12BE\u12C0\u12C2-\u12C5\u12C8-\u12D6\u12D8-\u1310\u1312-\u1315\u1318-\u135A\u1380-\u138F\u13A0-\u13F4\u1401-\u166C\u166F-\u167F\u1681-\u169A\u16A0-\u16EA\u1700-\u170C\u170E-\u1711\u1720-\u1731\u1740-\u1751\u1760-\u176C\u176E-\u1770\u1780-\u17B3\u17D7\u17DC\u1820-\u1877\u1880-\u18A8\u18AA\u18B0-\u18F5\u1900-\u191C\u1950-\u196D\u1970-\u1974\u1980-\u19AB\u19C1-\u19C7\u1A00-\u1A16\u1A20-\u1A54\u1AA7\u1B05-\u1B33\u1B45-\u1B4B\u1B83-\u1BA0\u1BAE\u1BAF\u1BBA-\u1BE5\u1C00-\u1C23\u1C4D-\u1C4F\u1C5A-\u1C7D\u1CE9-\u1CEC\u1CEE-\u1CF1\u1CF5\u1CF6\u1D00-\u1DBF\u1E00-\u1F15\u1F18-\u1F1D\u1F20-\u1F45\u1F48-\u1F4D\u1F50-\u1F57\u1F59\u1F5B\u1F5D\u1F5F-\u1F7D\u1F80-\u1FB4\u1FB6-\u1FBC\u1FBE\u1FC2-\u1FC4\u1FC6-\u1FCC\u1FD0-\u1FD3\u1FD6-\u1FDB\u1FE0-\u1FEC\u1FF2-\u1FF4\u1FF6-\u1FFC\u2071\u207F\u2090-\u209C\u2102\u2107\u210A-\u2113\u2115\u2119-\u211D\u2124\u2126\u2128\u212A-\u212D\u212F-\u2139\u213C-\u213F\u2145-\u2149\u214E\u2183\u2184\u2C00-\u2C2E\u2C30-\u2C5E\u2C60-\u2CE4\u2CEB-\u2CEE\u2CF2\u2CF3\u2D00-\u2D25\u2D27\u2D2D\u2D30-\u2D67\u2D6F\u2D80-\u2D96\u2DA0-\u2DA6\u2DA8-\u2DAE\u2DB0-\u2DB6\u2DB8-\u2DBE\u2DC0-\u2DC6\u2DC8-\u2DCE\u2DD0-\u2DD6\u2DD8-\u2DDE\u2E2F\u3005\u3006\u3031-\u3035\u303B\u303C\u3041-\u3096\u309D-\u309F\u30A1-\u30FA\u30FC-\u30FF\u3105-\u312D\u3131-\u318E\u31A0-\u31BA\u31F0-\u31FF\u3400-\u4DB5\u4E00-\u9FCC\uA000-\uA48C\uA4D0-\uA4FD\uA500-\uA60C\uA610-\uA61F\uA62A\uA62B\uA640-\uA66E\uA67F-\uA697\uA6A0-\uA6E5\uA717-\uA71F\uA722-\uA788\uA78B-\uA78E\uA790-\uA793\uA7A0-\uA7AA\uA7F8-\uA801\uA803-\uA805\uA807-\uA80A\uA80C-\uA822\uA840-\uA873\uA882-\uA8B3\uA8F2-\uA8F7\uA8FB\uA90A-\uA925\uA930-\uA946\uA960-\uA97C\uA984-\uA9B2\uA9CF\uAA00-\uAA28\uAA40-\uAA42\uAA44-\uAA4B\uAA60-\uAA76\uAA7A\uAA80-\uAAAF\uAAB1\uAAB5\uAAB6\uAAB9-\uAABD\uAAC0\uAAC2\uAADB-\uAADD\uAAE0-\uAAEA\uAAF2-\uAAF4\uAB01-\uAB06\uAB09-\uAB0E\uAB11-\uAB16\uAB20-\uAB26\uAB28-\uAB2E\uABC0-\uABE2\uAC00-\uD7A3\uD7B0-\uD7C6\uD7CB-\uD7FB\uF900-\uFA6D\uFA70-\uFAD9\uFB00-\uFB06\uFB13-\uFB17\uFB1D\uFB1F-\uFB28\uFB2A-\uFB36\uFB38-\uFB3C\uFB3E\uFB40\uFB41\uFB43\uFB44\uFB46-\uFBB1\uFBD3-\uFD3D\uFD50-\uFD8F\uFD92-\uFDC7\uFDF0-\uFDFB\uFE70-\uFE74\uFE76-\uFEFC\uFF21-\uFF3A\uFF41-\uFF5A\uFF66-\uFFBE\uFFC2-\uFFC7\uFFCA-\uFFCF\uFFD2-\uFFD7\uFFDA-\uFFDC-\u0400-\u04FF]+)/g;
    return text.match(regex) ? 1 : 0;
}


feat(*) Remove jitter with plural events and expected events

There's one or two more TODOs that I've outlined below, but I'm marking as ready for review to get some input around the rest of the methodology.
This PR:

[x] Adds support for ActorsStarted / ActorsStopped and their failure events
[x] Implicitly requires wadm to work with v0.63+ versions of the host because of the above
[x] Removes the notion of a backoff, replacing it with logic to make Scalers compute "expected events" for each command that they send. For example, if a scaler wants to start 5 of an actor, it won't take any action until it either receives an event where 5 of that actor started, or if that command failed to complete. Scaler implementers do not need to care about this, it's all on the wadm side.
[ ] TODO: Make the expected events list clear after a certain time frame to account for missed events or strange edge cases.

Feature or Problem
Currently, Scalers (particularly the ActorSpreadScaler) are prone to jitter and difficulty actually converging on proper state when you run many (10+) copies of an actor. They are handling each event as it comes in, and we added a very coarse backoff mechanism to account for this but it still creates difficulties.
Related Issues
Will file issue soon to make state only update on plural events.
Release Information
next alpha release
Consumer Impact
Consumers will notice that wadm feels more responsive at reconciling
Testing

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin
[ ] x86_64-windows

Tested on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin
[ ] x86_64-windows

Unit Test(s)
This feature is large enough that it probably warrants some tests to verify behavior around expected events.

[ ] TODO: Tests

Acceptance or Integration

Manual Verification
I've applied applications and messed with them in many ways to confirm that this works great with 1 running wadm.

Pretty positive I handled the rebase properly, but if I removed something from #118 lmk


Bump omero ice ansible role to 2.1.1

because of https://github.com/openmicroscopy/ansible-role-ice/issues/4

Thanks.


Cron job does not get updated on cron file changes

If I start a cron job using a specific cron file it works but the cron job doesn't update its information if I edit and save with new modifications in the cron file.

That’s correct. You can send Supercronic a SIGUSR2 signal if you want it to
reload your crontab.
On Sun, 19 Jul 2020 at 21:46, Codrin<EMAIL_ADDRESS>wrote:

If I start a cron job using a specific cron file it works but the cron job
doesn't update its information if I edit and save with new modifications in
the cron file.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/aptible/supercronic/issues/73, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANIHVRLRD65ZKLPGP4XUNDR4NLRRANCNFSM4PBSEE6Q
.


[feature request] make snow layers act like carpets on stairs

It's long bothered me that when  you use stairs to make roofs in wintery climates, the flat bits of the roofs will gather snow, but the stairs don't.   1 snow layer is almost identical to a carpet, so shouldn't be much of a stretch for this mod to add.

I have no plans to rework how Minecraft snowfall or snow generation works at the moment. Note now layers have the additional difficulty of being able to stack up to 8 layers, which raises the question of if further layers stacked on top should also be depressed.

there's already a mod for this!
https://www.curseforge.com/minecraft/mc-mods/snow-real-magic
no neeed for it to be added here!


[enhancement]: Emulate the automatic1111 remote API

Is there an existing issue for this?

[X] I have searched the existing issues

Contact Details
No response
What should this feature add?
There is an emerging ecosystem of integrations like https://github.com/tgiesselmann/gimp-stable-boy that use Automatic1111’s remote API (when it is launched with —api) to invoke Stable Diffusion. For artists who prefer InvokeAI but also use tools like GIMP or Krita or others (on a growing list of emerging plugins written against this API) in their usual workflow it would be ideal if the API could be emulated rather than requiring setup and care of two separate Stable Diffusion derivative installations.
Alternatives
No response
Aditional Content
No response

My thoughts exactly. Bump!

Bump! I also thought of this!

@CapableWeb , perhaps you'd be interested in working on this?

Yeah, this would be a solid addition.
Currently one of the reasons I'm relying on A1111.

We are approaching migration to a new, far more powerful web API which will make development of plugins and additional clients much more straightforward. However, I do not see a future in which we emulate or provide compatibility with A1111 as we will be processing requests in a very different way.

@psychedelicious is this a REST API?  Or WebSocket based one?
The Blender dream-textures extension is currently using a fork of InvokeAI but it'd be cool if it could plug into an existing instance.  However I wasn't able to find web API documentation anywhere, I guess that's coming soon?
(See here for relevant dream-textures issue: https://github.com/carson-katri/dream-textures/issues/225)

@micahscopes
It's mostly HTTP with some signaling (like generation progress) done via socket.io.
It uses a graph execution system, where you send json describing processing nodes and the links between them. See #1650 for detail.
We will provide proper support and documentation for this new api (current one was built alongside the web ui and not intended for public consumption).
I'm not sure of an eta but I am preparing the web app for migration to the new system now.

@psychedelicious
Do these graphs more or less map directly to diffusers pipelines? The dream-textures library has switched to using diffusers directly and are passing around pipeline descriptions, so I'm curious to understand if their new execution system could be mapped to your upcoming API.

@micahscopes
No, the graphs are a layer or two above diffusers pipelines (in fact we have just migrated to using diffusers ourselves).
A simple example graph is: txt2img node --> img2img node --> showImage node
In this simple case, they are linked serially, but so long as inputs and outputs are of the same type, any nodes can be linked. When this graph is executed, an image is generated via txt2img and its image output is fed into img2img, generating a new image, which then is fed into showImage, which shows the image to the user.


Update Link widget

Update:
https://github.com/creativecreatorormaybenot/funvas/blob/d8f815dd8965dc46b1567510c7eb9787ce9e29d5/funvas_demo/lib/widgets/link.dart#L6
To the one used in GitHub tracker (uses anchor element and web-like styling in general).

(also bump url_launcher)

https://github.com/creativecreatorormaybenot/github-tracker/blob/406c99c0f9ee04bf4c8f6fcaa351ba79b02ea085/frontend/github_tracker/lib/widgets/link.dart#L10


Problem of The table.2

In the paper Table.2  the result of MLP-LINEAR and  MLP-3, I want to know the number of runs to get the mean and variance. Thank you！

Hi @kengkeng1,
We typically run 5 times.


KAFKA-2464 [WIP]: client-side assignment for new consumer

I left a few comments, but mostly they were things that could potentially be clarified. Only one question about correctness. After those issues are followed up on, this looks good to me.

@ewencp @guozhangwang @onurkaraman I've updated this patch based on the latest proposal (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal), which uses a single client to do the group assignment. I'm still doing a bit of polishing and adding tests, but since most existing tests are passing, it seems ready to start reviewing. There is one failing copycat test that I have yet to investigate.
A few general notes on the patch:

One significant difference from the proposal is that I had to push the assignment strategies out of the consumer's embedded protocol and into the JoinGroup request as a "subProtocols" field. This allows the coordinator to do assignment strategy selection similar to the initial patch. It also lets the coordinator reject incompatible members without forcing a group rebalance. Otherwise, we can only detect inconsistencies after the group has rebalanced and the leader sees the metadata from all members. The problem then becomes how the group should handle the inconsistency? We could let the leader kick certain members out of the group, but that complicates the protocol. In the end, it seemed simplest to give the coordinator a mechanism to validate compatibility of group members.
This brings up the issue of whether each "subProtocol" (or assignment strategy) needs to have its own metadata and embedded as in the initial patch. Without it, any assignment-strategy-specific information must itself be embedded in the member metadata included in the JoinGroup request. This is easy if the consumer is using a metadata format like JSON, but trickier with packed arrays. The advantage of keeping everything in the same metadata, however, is that it simplifies the protocol and avoids redundancies such as including subscriptions multiple times.
If the leader has an unexpected error during synchronization, the patch does not provide a way to propagate that error to the rest of the group as the proposal called for. Instead, the leader will propagate the exception to the user and the leader's session timeout will expire on the coordinator which will allow for a new leader to be elected. The problem I found when trying to implement this feature is that it requires a new state on the coordinator which is basically like Stable except contains an error to be propagated when the rest of the group syncs, which adds considerable complexity. It also creates a race condition where some members might see the error and immediately rejoin which forces the group out of that state. This means that some members wouldn't generally see the error. In the end, it felt like the cost of propagating the error through the coordinator was too high to justify capability it was exposing. Since unexpected leader failures should be rare, the simpler approach is just to let it timeout.
The versioning in the consumer's embedded protocol (as contained in ConsumerProtocol.java) is clearly inadequate and will require further consideration. I'm inclined to push that to a separate JIRA so that we can consider it without the rest of the noise around this patch.
The protocol implemented here provides no special support for regex handling. As before, each member watches metadata and adds a normal topic subscription when it finds a new matching topic. If we want to have special handling, that can also be pushed to another JIRA.

@hachikuji Not critical for this patch, but do we want to reconsider the packages for some of the classes? For example, GroupCoordinator is currently in org.apache.kafka.clients.consumer.internals. Keeping it in some internals package makes sense at the moment to indicate it shouldn't be relied on publicly, but since it's generalized functionality, housing it under consumer might not be ideal. Not sure if we want a new package (e.g. org.apache.kafka.clients.group), move it under common, or just leave it where it is and preserve some indication of its heritage.

Just a heads-up: this branch doesn't merge cleanly against trunk anymore.

@ewencp Yeah, I think it makes sense to move it to a new package. To make that possible, I think we might have to merge ConsumerNetworkClient into NetworkClient, but that's the only complication I can think of.

A general comment:
I feel adding this to the join-group to allow coordinator-side pre-checking would be OK, since although it adds an extra field in the join group request, it is not very hard to understand. But now the protocol field seems less useful to me given the group-id and sub-protocols, since we only use it to check all members use the same group-protocol string. But in practice how possible that members joining the same group name happen to be different types (e.g. one normal consumer and one copycat connector)? Having this field will not be so useful in excluding human-mistakes but even likely increase false-positives that two members that do belong to the same group got kicked out because they happen to specify different protocol names by mistake.
So I would suggest we remove the protocol string and rename sub-protocols to protocols, doing this we may end up having group ids like "kafka-consumer-xxx", "kafka-stream-yyy" and "kafka-copycat-zzz", which is already sufficient to differentiate different types of members.

@guozhangwang That suggestion sounds fine to me, especially if it makes the protocol easier to understand. It leads to the question of how we are going to implement protocol-specific metadata. There's basically two options:
Option 1: Each protocol has its own metadata: this is probably the simplest to implement for client extensions. This would change the JoinGroup request to look something like this:
JoinGroupRequest => GroupId SessionTimeout MemberId Protocols
  GroupId => String
  SessionTimeout => int32
  MemberId => String
  Protocols => [Protocol MemberMetadata]
    Protocol => String
    MemberMetadata => Bytes

The downside is that this leads to redundant data in the request. For example, to support both the round-robin and range assignors in the consumer (at the same time), you'd have to pass the subscription list twice. Perhaps this is a rare enough case that we aren't too worried about the overhead? It also leads to somewhat more complex parsing, though I don't think this is a major concern.
Option 2: The second option, which is currently implemented, is that we have only one metadata field and we assume that any protocol-specific attributes are embedded in it. It would look like this:
JoinGroupRequest => GroupId SessionTimeout MemberId Protocols MemberMetadata
  GroupId => String
  SessionTimeout => int32
  MemberId => String
  Protocols => [String]
  MemberMetadata => Bytes

This is generally more difficult for clients to implement, but that depends largely on the format of the metadata. For example, if JSON is chosen, then protocol-specific attributes come naturally. However, it's a little tough to see how this would work for Kafka's packed structures, which have no field identifiers. As an advantage, since it allows protocols to share metadata, it avoids the duplication in the first approach.
The GroupProtocol field in the current patch was basically intended to declare compatibility of the metadata as used in Option 2. If we get rid of it, then I would tend to favor the first option. What do you think?

It might be worth making multiple assignors work to figure out how best to handle this. Right now the code only handles one assignor and I think that makes option 2 look like it can work, but the code is probably going to get messy or confusing if you try to actually implement it. Since you only have one metadata field, you'll need to somehow combine all the metadata required by different assignors. This means you'll have to assume some format like JSON and merge them.
I think the duplication in Option 1 isn't a big deal. It should definitely be rare. It also keeps things simpler -- the JoinGroup request is a bit more complex, but the whole implementation is easier and we don't end up with completely different protocols having to coordinate in order to make sure their metadata is compatible with each other.

My understanding is that multiple protocols will only be used for upgrades, and most of time the protocols size should be 1. If that is the case:

98% we will have 1 protocol.
1.9999% we will have 2 protocols.
0.0001% we will have 3 protocols.

So I think option 1) would be find, but @ewencp could clarify if there are no scenarios that multiple protocols are needed life-time.

I think 3 protocols should never happen aside from misconfiguration. The only use case for 2 is to upgrade/switch assignors.

Yeah, for upgrade / switch 2 protocols are temporary since we will usually do two rolling bounce to remove the old one in the second round.
Anyways, I think option 1) should be fine.

@ewencp @guozhangwang I've reworked the protocol as discussed to allow protocol-specific metadata with coordinator negotiation similar to the initial patch. Basically this resulted in pushing serialization concerns into AbstractCoordinator extensions. Also have a look at the new PartitionAssignor interface. It should be much clearer how custom implementations can leverage custom metadata.

@ewencp @guozhangwang Made the changes discussed. I think this does simplify the assignor internals quite a bit.

@hachikuji The client-side protocol and implementations LGTM overall. Thanks!
Do you want to refactor the server-side FSM a bit now? Also some response error codes like SyncGroupResponse need to be updated as well.

@guozhangwang Anything else to clear up before this can be merged?

I'm still reviewing the changes made on the server-side coordinator, I thought you did the FSM refactoring right?

@guozhangwang  I updated the FSM documentation that was in GroupMetadata.scala. Was there anything else you wanted me to fix? By the way, I'm rebasing off of Flavio's commit and should be able to update shortly.

The problem is that I cannot get the full diff now as it become too big. Right now I can only review it per-commits which is not optimal..

This is what I think about the state machine (if we agree we can also update the diagram in the wiki), with the assumption that for all request handling we first check coordinator availability and group ownership is correct. All the events except consumer failure is triggered by requests.
State Down: group is either not created or has no members.
-> onJoin: create group and add member if necessary (start HB in purgatory), transit to PrepareRebalance.
-> onSync: return REBALANCE_IN_PROGRESS (could be from AS -> PR -> Down).
-> onHeartBeat: return ILLEGAL_GENERATION.
-> onOffsetCommit: blindly accepts and return OK.
-> onOffsetFetch: blindly accepts and return OK.
-> onConsumerFailure: SHOULD NOT HAPPEN, throw exception.
State PrepareRebalance: waiting for all members to Join
-> onJoin: check if all members joined, if yes transit to AwaitingSync
-> onSync: return REBALANCE_IN_PROGRESS (could be from AS -> PR).
-> onHeartBeat: return REBALANCE_IN_PROGRESS.
-> onOffsetCommit: return REBALANCE_IN_PROGRESS.
-> onOffsetFetch: return REBALANCE_IN_PROGRESS.
-> onConsumerFailure: remove consumer (stop HB in purgatory), check if all rest members joined, if yes transit to AwaitingSync.
State AwaitingSync: waiting for leader to assign
-> onJoin: update / create consumer if necessary, transit to PrepareRebalance
-> onSync: if from follower, treat it as HB and park it (unmute further HBs); otherwise treat it as HB for the leader and transit to Stable.
-> onHeartBeat: return REBALANCE_IN_PROGRESS.
-> onOffsetCommit: return REBALANCE_IN_PROGRESS.
-> onOffsetFetch: return REBALANCE_IN_PROGRESS.
-> onConsumerFailure: remove consumer (stop HB in purgatory), if that is the leader transit to PrepareRebalance
State Stable: group formed with resource assigned and remembered by Coordinator
-> onJoin: update / create consumer if necessary, transit to PrepareRebalance
-> onSync: handle normally with remembered assignment
-> onHeartBeat: handle normally.
-> onOffsetCommit: check on member-id / generation-id and return OK or corresponding errors.
-> onOffsetFetch: check on member-id / generation-id and return OK or corresponding errors.
-> onConsumerFailure: remove consumer (stop HB in purgatory), transit to PrepareRebalance.

Transit action: from Down to PrepareRebalance:
1) start delayed rebalance in purgatory if there are still members in the group
2) mute delayed HB for all members in the group
Transit action: from PrepareRebalance to AwaitingSync:
1) clear dalyed rebalance in purgatory, send back Join response to followers and leader
2) unmute delayed HB for all members in the group
Transit action: from AwaitingSync to PrepareRebalance:
1) send back Sync response to all parked requests with REBALANCE_IN_PROGRESS
2) start delayed rebalance in purgatory if there are still members in the group
3) mute delayed HB for all members in the group
Transit action: from Stable to PrepareRebalance:
1) start delayed rebalance in purgatory if there are still members in the group
2) mute delayed HB for all members in the group

I think we are already doing this in this patch, but just want to make the comments in GroupMetadata to be more clear.
And not need to include in this JIRA, but moving forward we will possibly add a background scheduler that periodically remove groups with zero members or not belonged to coordinator any more and transit them to Dead.

@guozhangwang Thanks for writing that up! I think there's a couple slight differences in the current patch from what you wrote:
State Down: group is either not created or has no members.
-> onHeartBeat: return UNKNOWN_MEMBER_ID
-> onOffsetCommit: return ILLEGAL_GENERATION if generation is positive, otherwise accept commit
-> onOffsetFetch: return UNKNOWN_MEMBER_ID
State PrepareRebalance: waiting for all members to Join
-> onOffsetCommit: accept commits from previous generation, otherwise ILLEGAL_GENERATION
-> onOffsetFetch: return offsets blindly
State AwaitingSync: waiting for leader to assign
-> onOffsetCommit: allow commit from the joined generation
-> onOffsetFetch: return offsets blindly
For the AwaitingSync state, I think your suggestion to return REBALANCE_IN_PROGRESS for offset commits is better than the current behavior since there should be no need to commit offsets before receiving the assignment and the member still has the PreparingRebalance state to commit offsets before rebalances.
For fetching committed offsets, I actually wonder if we should skip all group checks and return the offsets blindly. Do we really need to impose the restriction that committed offsets can only be queried by members of the group?

@hachikuji Could we make partitioner also Configurable as we discussed offline? Besides the above comments LGTM.

@guozhangwang Have a look at the updated state transitions. I wonder if we should have a Ready state for the initial state when no group members have joined. Currently the group starts up in Stable even though it has no members.
@onurkaraman What do you think?

I agree to the Ready state, since if we are going to clean up groups moving forward we will no longer need the Dead state any more.

LGTM, the Jenkins failures seem irrelevant and transient. Also made a passing Ducktape system test build on this branch. @hachikuji Please remember to address @becketqin 's comments in the later follow-up path. Merging this large patch for now as it is getting very large.