Different Initial condition for the cells in pack

I tried to populate different initial electrolyte concentration across the cells in the pack. I have an array of initial electrolyte concentration (initial_eleConc) similar to total heat transfer coefficient (htc). The changes I made to utils.py, simulations.py, solver_utils.py are given below.
Changed line 220, and added line 229 in utils.py.
# -*- coding: utf-8 -*-
"""
Created on Thu Sep 23 10:33:13 2021

@author: Tom
"""
from scipy.interpolate import interp1d, interp2d
import numpy as np
import pandas as pd
import os
import liionpack
from skspatial.objects import Plane
from skspatial.objects import Points

ROOT_DIR = os.path.dirname(os.path.abspath(liionpack.__file__))
CIRCUIT_DIR = os.path.join(ROOT_DIR, "circuits")
DATA_DIR = os.path.join(ROOT_DIR, "data")
INIT_FUNCS = os.path.join(ROOT_DIR, "init_funcs")

def interp_current(df):
    r"""
    Returns an interpolation function for current w.r.t time

    Parameters
    ----------
    df : pandas.DataFrame or Dict
        Contains data for 'Time' and 'Cells Total Current' from which to
        construct an interpolant function

    Returns
    -------
    f : function
        interpolant function of total cell current with time.

    """
    t = df["Time"]
    I = df["Cells Total Current"]
    f = interp1d(t, I)
    return f

def _z_from_plane(X, Y, plane):
    r"""
    Given X and Y and a plane provide Z
    X - temperature
    Y - flow rate
    Z - heat transfer coefficient

    Parameters
    ----------
    X : float (array)
        x-coordinate.
    Y : float (array)
        z-coordinate.
    plane : skspatial.object.Plane
        plane returned from read_cfd_data.

    Returns
    -------
    z : float (array)
        z-coordinate.

    """
    a, b, c = plane.normal
    d = plane.point.dot(plane.normal)
    z = (d - a * X - b * Y) / c
    return z

def _fit_plane(xv, yv, dbatt):
    r"""
    Private method to fit plane to CFD data

    Parameters
    ----------
    xv : ndarray
        temperature meshgrid points.
    yv : ndarray
        flow_rate meshgrid points.
    dbatt : ndarray
        cfd data for heat transfer coefficient.

    Returns
    -------
    plane : skspatial.object.Plane
        htc varies linearly with temperature and flow rate so relationship
        describes a plane

    """
    nx, ny = xv.shape
    pts = []
    for i in range(nx):
        for j in range(ny):
            pts.append([xv[i, j], yv[i, j], dbatt[i, j]])

    points = Points(pts)
    plane = Plane.best_fit(points, tol=1e-6)
    return plane

def read_cfd_data(data_dir=None, filename="cfd_data.xlsx", fit="linear"):
    r"""
    A very bespoke function to read heat transfer coefficients from an excel
    file

    Parameters
    ----------
    data_dir : str, optional
        Path to data file. The default is None. If unspecified the module
        liionpack.DATA_DIR folder will be used
    filename : str, optional
        The default is 'cfd_data.xlsx'.
    fit : str
        options are 'linear' (default) and 'interpolated'.
    Returns
    -------
    funcs : list
        an interpolant is returned for each cell in the excel file.

    """
    if data_dir is None:
        data_dir = liionpack.DATA_DIR
    fpath = os.path.join(data_dir, filename)
    ncells = 32
    flow_bps = np.array(pd.read_excel(fpath, sheet_name="massflow_bps", header=None))
    temp_bps = np.array(pd.read_excel(fpath, sheet_name="temperature_bps", header=None))
    xv, yv = np.meshgrid(temp_bps, flow_bps)
    data = np.zeros([len(temp_bps), len(flow_bps), ncells])
    fits = []
    for i in range(ncells):
        data[:, :, i] = np.array(
            pd.read_excel(fpath, sheet_name="cell" + str(i + 1), header=None)
        )
        # funcs.append(interp2d(xv, yv, data[:, :, i], kind='linear'))
        if fit == "linear":
            fits.append(_fit_plane(xv, yv, data[:, :, i]))
        elif fit == "interpolated":
            fits.append(interp2d(xv, yv, data[:, :, i], kind="linear"))

    return data, xv, yv, fits

def get_linear_htc(planes, T, Q):
    r"""
    A very bespoke function that is called in the solve process to update the
    heat transfer coefficients for every battery - assuming linear relation
    between temperature, flow rate and heat transfer coefficient.

    Parameters
    ----------
    planes : list
        each element of the list is a plane equation describing linear relation
        between temperature, flow rate and heat transfer coefficient.
    T : float array
        The temperature of each battery.
    Q : float
        The flow rate for the system.

    Returns
    -------
    htc : float
        Heat transfer coefficient for each battery.

    """
    ncell = len(T)
    htc = np.zeros(ncell)
    for i in range(ncell):
        htc[i] = _z_from_plane(T[i], Q, planes[i])
    return htc

def get_interpolated_htc(funcs, T, Q):
    r"""
    A very bespoke function that is called in the solve process to update the
    heat transfer coefficients for every battery

    Parameters
    ----------
    funcs : list
        each element of the list is an interpolant function.
    T : float array
        The temperature of each battery.
    Q : float
        The flow rate for the system.

    Returns
    -------
    htc : float
        Heat transfer coefficient for each battery.

    """
    ncell = len(T)
    htc = np.zeros(ncell)
    for i in range(ncell):
        htc[i] = funcs[i](T[i], Q)
    return htc

def build_inputs_dict(I_batt, htc,initial_eleConc):
    r"""
    Function to convert inputs and external_variable arrays to list of dicts
    As expected by the casadi solver. These are then converted back for mapped
    solving but stored individually on each returned solution.
    Can probably remove this process later

    Parameters
    ----------
    I_batt : float array
        The input current for each battery.
    htc : float array
        the heat transfer coefficient for each battery.

    Returns
    -------
    inputs_dict : list
        each element of the list is an inputs dictionary corresponding to each
        battery.

    """
    inputs_dict = []
    for i in range(len(I_batt)):
        inputs_dict.append(
            {
                # 'Volume-averaged cell temperature': T_batt[i],
                "Current": I_batt[i],
                "Total heat transfer coefficient [W.m-2.K-1]": htc[i],
                "Initial concentration in electrolyte [mol.m-3]": initial_eleConc[i],
            }
        )
    return inputs_dict

Added line 73 in simulations.py.
# -*- coding: utf-8 -*-
"""
Created on Wed Sep 22 15:37:51 2021

@author: tom
"""

import pybamm

def _current_function(t):
    r"""
    Internal function to make current an input parameter

    Parameters
    ----------
    t : float
        Dummy time parameter.

    Returns
    -------
    TYPE
        DESCRIPTION.

    """
    return pybamm.InputParameter("Current")

def create_simulation(parameter_values=None, experiment=None, make_inputs=False):
    r"""
    Create a PyBaMM simulation set up for interation with liionpack

    Parameters
    ----------
    parameter_values : pybamm.ParameterValues class
        DESCRIPTION. The default is None.
    experiment : pybamm.Experiment class
        DESCRIPTION. The default is None.
    make_inputs : bool, optional
        Changes "Current function [A]" and "Total heat transfer coefficient
        [W.m-2.K-1]" to be inputs that are controlled by liionpack.
        The default is False.

    Returns
    -------
    sim : pybamm.Simulation
        A simulation that can be solved individually or passed into the
        liionpack solve method

    """
    # Create the pybamm model
    model = pybamm.lithium_ion.SPMe(
        options={
            "thermal": "lumped",
        }
    )
    # geometry = model.default_geometry
    if parameter_values is None:
        # load parameter values and process model and geometry
        chemistry = pybamm.parameter_sets.Chen2020
        parameter_values = pybamm.ParameterValues(chemistry=chemistry)
    # Change the current function to be an input as this is set by the external circuit
    if make_inputs:
        parameter_values.update(
            {
                "Current function [A]": _current_function,
            }
        )
        parameter_values.update(
            {
                "Current": "[input]",
                "Total heat transfer coefficient [W.m-2.K-1]": "[input]",
                "Initial concentration in electrolyte [mol.m-3]":"[input]",
            },
            check_already_exists=False,
        )

    solver = pybamm.CasadiSolver(mode="safe")
    sim = pybamm.Simulation(
        model=model,
        experiment=experiment,
        parameter_values=parameter_values,
        solver=solver,
    )
    return sim

if __name__ == "__main__":
    sim = create_simulation()
    sim.solve([0, 1800])
    sim.plot()

Changed lines 86, 122,163,256,266 in solver_utils.py.
# -*- coding: utf-8 -*-
"""
Created on Thu Sep 23 10:44:31 2021

@author: Tom
"""
import casadi
import pybamm
import numpy as np
import time as ticker
import liionpack as lp

def _mapped_step(model, solutions, inputs_dict, integrator, variables, t_eval):
    r"""
    Internal function to process the model for one timestep in a mapped way.
    Mapped versions of the integrator and variables functions should already
    have been made.

    Parameters
    ----------
    model : pybamm.Model
        The built model
    solutions : list of pybamm.Solution objects for each battery
        Used to get the last state of the system and use as x0 and z0 for the
        casadi integrator
    inputs_dict : list of inputs_dict objects for each battery
        DESCRIPTION.
    integrator : mapped casadi.integrator
        Produced by _create_casadi_objects
    variables : mapped variables evaluator
        Produced by _create_casadi_objects
    t_eval : float array of times to evaluate
        Produced by _create_casadi_objects

    Returns
    -------
    sol : list
        solutions that have been stepped forward by one timestep
    var_eval : list
        evaluated variables for final state of system

    """
    len_rhs = model.concatenated_rhs.size
    N = len(solutions)
    if solutions[0] is None:
        # First pass
        x0 = casadi.horzcat(*[model.y0[:len_rhs] for i in range(N)])
        z0 = casadi.horzcat(*[model.y0[len_rhs:] for i in range(N)])
    else:
        x0 = casadi.horzcat(*[sol.y[:len_rhs, -1] for sol in solutions])
        z0 = casadi.horzcat(*[sol.y[len_rhs:, -1] for sol in solutions])
    # t_min = [0.0]*N
    t_min = 0.0
    inputs = []
    for temp in inputs_dict:
        inputs.append(casadi.vertcat(*[x for x in temp.values()] + [t_min]))
    ninputs = len(temp.values())
    inputs = casadi.horzcat(*inputs)
    # p = casadi.horzcat(*zip(inputs, external_variables, [t_min]*N))
    # inputs_with_tmin = casadi.vertcat(inputs, np.asarray(t_min))
    # Call the integrator once, with the grid
    timer = pybamm.Timer()
    tic = timer.time()
    casadi_sol = integrator(x0=x0, z0=z0, p=inputs)
    integration_time = timer.time()
    nt = len(t_eval)
    xf = casadi_sol["xf"]
    # zf = casadi_sol["zf"]
    sol = []
    xend = []
    for i in range(N):
        start = i * nt
        y_sol = xf[:, start:start + nt]
        xend.append(y_sol[:, -1])
        # Not sure how to index into zf - need an example
        sol.append(pybamm.Solution(t_eval, y_sol, model, inputs_dict[i]))
        sol[-1].integration_time = integration_time
    toc = timer.time()
    lp.logger.debug(f"Mapped step completed in {toc - tic}")
    xend = casadi.horzcat(*xend)
    var_eval = variables(0, xend, 0, inputs[0:ninputs, :])
    return sol, var_eval

def _create_casadi_objects(I_init, htc,initial_eleConc, sim, dt, Nspm, nproc, variable_names):
    r"""
    Internal function to produce the casadi objects in their mapped form for
    parallel evaluation

    Parameters
    ----------
    I_init : float
        initial guess for current of a battery (not used for simulation).
    htc : float
        initial guess for htc of a battery (not used for simulation).
    sim : pybamm.Simulation
        A PyBaMM simulation object that contains the model, parameter_values,
        solver, solution etc.
    dt : float
        The time interval for a single timestep. Fixed throughout the simulation
    Nspm : int
        Number of individual batteries in the pack.
    nproc : int
        Number of parallel processes to map to.
    variable_names : list
        Variables to evaluate during solve. Must be a valid key in the
        model.variables

    Returns
    -------
    integrator : mapped casadi.integrator
        Solves an initial value problem (IVP) coupled to a terminal value
        problem with differential equation given as an implicit ODE coupled
        to an algebraic equation and a set of quadratures
    variables_fn : mapped variables evaluator
        evaluates the simulation and output variables. see casadi function
    t_eval : float array of times to evaluate
        times to evaluate in a single step, starting at zero for each step

    """
    inputs = {"Current": I_init, "Total heat transfer coefficient [W.m-2.K-1]": htc,"Initial concentration in electrolyte [mol.m-3]": initial_eleConc}
    solver = sim.solver
    # solve model for 1 second to initialise the circuit
    t_eval = np.linspace(0, 1, 2)
    # Initial solution - this builds the model behind the scenes
    sim.solve(t_eval, inputs=inputs)
    # step model
    # Code to create mapped integrator
    t_eval = np.linspace(0, dt, 11)
    t_eval_ndim = t_eval / sim.model.timescale.evaluate()
    inp_and_ext = inputs
    # No external variables - Temperature solved as lumped model in pybamm
    # External variables could (and should) be used if battery thermal problem
    # Includes conduction with any other circuits or neighboring batteries
    # inp_and_ext.update(external_variables)

    integrator = solver.create_integrator(
        sim.built_model, inputs=inp_and_ext, t_eval=t_eval_ndim
    )
    integrator = integrator.map(Nspm, "thread", nproc)

    # Variables function for parallel evaluation
    casadi_objs = sim.built_model.export_casadi_objects(variable_names=variable_names)
    variables = casadi_objs["variables"]
    t, x, z, p = (
        casadi_objs["t"],
        casadi_objs["x"],
        casadi_objs["z"],
        casadi_objs["inputs"],
    )
    variables_stacked = casadi.vertcat(*variables.values())
    variables_fn = casadi.Function("variables", [t, x, z, p], [variables_stacked])
    variables_fn = variables_fn.map(Nspm, "thread", nproc)
    return integrator, variables_fn, t_eval

def solve(
    netlist=None,
    parameter_values=None,
    experiment=None,
    I_init=1.0,
    htc=None,initial_eleConc=None,
    initial_soc=0.5,
    nproc=12,
    output_variables=None,
):
    r"""
    Solves a pack simulation

    Parameters
    ----------
    netlist : pandas.DataFrame
        A netlist of circuit elements with format. desc, node1, node2, value.
        Produced by liionpack.read_netlist or liionpack.setup_circuit
    parameter_values : pybamm.ParameterValues class
        A dictionary of all the model parameters
    experiment : pybamm.Experiment class
        The experiment to be simulated. experiment.period is used to
        determine the length of each timestep.
    I_init : float, optional
        Initial guess for single battery current [A]. The default is 1.0.
    htc : float array, optional
        Heat transfer coefficient array of length Nspm. The default is None.
    initial_soc : float
        The initial state of charge for every battery. The default is 0.5
    nproc : int, optional
        Number of processes to start in parallel for mapping. The default is 12.
    output_variables : list, optional
        Variables to evaluate during solve. Must be a valid key in the
        model.variables

    Raises
    ------
    Exception
        DESCRIPTION.

    Returns
    -------
    output : ndarray shape [# variable, # steps, # batteries]
        simulation output array

    """

    if netlist is None or parameter_values is None or experiment is None:
        raise Exception("Please supply a netlist, paramater_values, and experiment")

    # Get netlist indices for resistors, voltage sources, current sources
    Ri_map = netlist["desc"].str.find("Ri") > -1
    V_map = netlist["desc"].str.find("V") > -1
    I_map = netlist["desc"].str.find("I") > -1

    Nspm = np.sum(V_map)

    protocol = lp.generate_protocol_from_experiment(experiment)
    dt = experiment.period
    Nsteps = len(protocol)

    # Solve the circuit to initialise the electrochemical models
    V_node, I_batt = lp.solve_circuit(netlist)

    sim = lp.create_simulation(parameter_values, make_inputs=True)
    lp.update_init_conc(sim, SoC=initial_soc)

    v_cut_lower = parameter_values["Lower voltage cut-off [V]"]
    v_cut_higher = parameter_values["Upper voltage cut-off [V]"]

    # The simulation output variables calculated at each step for each battery
    # Must be a 0D variable i.e. battery wide volume average - or X-averaged for 1D model
    variable_names = [
        "Terminal voltage [V]",
        "Measured battery open circuit voltage [V]",
        "Local ECM resistance [Ohm]",
    ]
    if output_variables is not None:
        for out in output_variables:
            if out not in variable_names:
                variable_names.append(out)
        # variable_names = variable_names + output_variables
    Nvar = len(variable_names)
    # Storage variables for simulation data
    shm_i_app = np.zeros([Nsteps, Nspm], dtype=float)
    shm_Ri = np.zeros([Nsteps, Nspm], dtype=float)
    output = np.zeros([Nvar, Nsteps, Nspm], dtype=float)

    # Initialize currents in battery models
    shm_i_app[0, :] = I_batt * -1

    time = 0
    # step = 0
    end_time = dt * Nsteps
    step_solutions = [None] * Nspm
    V_terminal = []
    record_times = []

    integrator, variables_fn, t_eval = _create_casadi_objects(
        I_init, htc[0],initial_eleConc[0], sim, dt, Nspm, nproc, variable_names
    )

    sim_start_time = ticker.time()

    for step in range(Nsteps):
        step_solutions, var_eval = _mapped_step(
            sim.built_model,
            step_solutions,
            lp.build_inputs_dict(shm_i_app[step, :], htc,initial_eleConc),
            integrator,
            variables_fn,
            t_eval,
        )
        output[:, step, :] = var_eval

        time += dt
        # Calculate internal resistance and update netlist
        temp_v = output[0, step, :]
        temp_ocv = output[1, step, :]
        temp_Ri = np.abs(output[2, step, :])
        shm_Ri[step, :] = temp_Ri

        netlist.loc[V_map, ("value")] = temp_ocv
        netlist.loc[Ri_map, ("value")] = temp_Ri
        netlist.loc[I_map, ("value")] = protocol[step]

        # print('Stepping time', np.around(ticker.time()-tic, 2), 's')
        if np.any(temp_v < v_cut_lower):
            print("Low V limit reached")
            break
        if np.any(temp_v > v_cut_higher):
            print("High V limit reached")
            break
        # step += 1
        if time <= end_time:
            record_times.append(time)
            V_node, I_batt = lp.solve_circuit(netlist)
            V_terminal.append(V_node.max())
        if time < end_time:
            shm_i_app[step + 1, :] = I_batt[:] * -1
    all_output = {}
    all_output["Time [s]"] = np.asarray(record_times)
    all_output["Pack current [A]"] = np.asarray(protocol[: step + 1])
    all_output["Pack terminal voltage [V]"] = np.asarray(V_terminal)
    all_output["Cell current [A]"] = shm_i_app[: step + 1, :]
    for j in range(Nvar):
        all_output[variable_names[j]] = output[j, : step + 1, :]

    toc = ticker.time()
    pybamm.logger.notice(
        "Solve circuit time " + str(np.around(toc - sim_start_time, 3)) + "s"
    )
    return all_output

The example code I ran is:
import liionpack as lp
import numpy as np
import pybamm

# Generate the netlist
netlist = lp.setup_circuit(Np=16, Ns=2, Rb=1e-4, Rc=1e-2, Ri=5e-2, V=3.2, I=80.0)

output_variables = [
    'X-averaged total heating [W.m-3]',
    'Volume-averaged cell temperature [K]',
    'X-averaged negative particle surface concentration [mol.m-3]',
    'X-averaged positive particle surface concentration [mol.m-3]',
    'X-averaged electrolyte concentration [mol.m-3]',
    ]

# Heat transfer coefficients
htc = np.ones(32) * 10
#initial_eleConc=np.ones(32) * 1000.0
initial_eleConc=np.array([1000., 1000., 2000., 2000., 1000., 1000., 1000., 1000., 1000.,
       1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000.,
       1000., 1000., 1000., 1000., 1000., 3000., 3000., 3000., 1000.,
       1000., 1000., 1000., 1000., 1000.])

# Cycling experiment, using PyBaMM
experiment = pybamm.Experiment(
    ["Charge at 50 A for 30 minutes", "Rest for 15 minutes", "Discharge at 50 A for 30 minutes", "Rest for 30 minutes"],
    period="10 seconds",
)

# PyBaMM parameters
chemistry = pybamm.parameter_sets.Chen2020
parameter_values = pybamm.ParameterValues(chemistry=chemistry)

# Solve pack
output = lp.solve(netlist=netlist,
                  parameter_values=parameter_values,
                  experiment=experiment,
                  output_variables=output_variables,
                  htc=htc,initial_eleConc=initial_eleConc)
lp.plot_output(output)

However, in the simulation the first value of initial electrolyte concentration, i.e., initial_eleConc[0] is populating across all the cells. Here is the output showing that

:
I think the issue is in line 256 of solver_utils.py, i.e.
integrator, variables_fn, t_eval = _create_casadi_objects(
        I_init, htc[0],initial_eleConc[0], sim, dt, Nspm, nproc, variable_names
    )

where it is only taking the first element of the array initial_eleConc[0].  I gave  initial_eleConc array similar to htc(heat transfer coefficient) array. As htc array is going into lumped model, so every cell can have their own individual heat transfer coefficient. In contrast, the parameters in the base model seems not populating across the cells.

Haven't looked specifically into what's happening here, but we should come up with a user interface that allows users to change any parameter like this, instead of adding each one as another argument  to all the functions

Haven't looked specifically into what's happening here, but we should come up with a user interface that allows users to change any parameter like this, instead of adding each one as another argument to all the functions

Agreed. The current has to be an input for Liionpack to work and is changed on every step. HTC technically doesn't but you would want it to be if solving a dynamic thermal problem and again might change a little bit for every step. The rest are more like ensemble constants and would be different to start with but then not change. Is it worth making this distinction and calling it something else?

Can you make a branch from main with the code changes you made?

I created a branch name issue-57-init_cond with the above changes

Hi @ksnvikrant this issue is also related to being able to pass in the model flexibly. I think the answer to both is to set up the simulation and then pass that into the liionpack solvers. Only problem there is how to enforce the current to be an input. I guess throwing errors if it's not might be ok for now. The code changes you made will now be out of date after the solver refactor so I think it's probably easier just to make them again off a new branch. I can combine this with the DFN model issue into a new branch

@ksnvikrant I've just pushed a bunch of changes which should help you with this issue. There is a new notebook on using inputs. Let me know if you have any questions

Hi @TomTranter Thanks for the changes. I tried to create an example for different initial SOC condition for the cells in pack. However, it is still taking the first initial SOC of the cell and populating across all other cells. In order to run the example below, first you need to comment line 44 of solvers.py, i.e., #lp.update_init_conc(self.simulation, SoC=initial_soc). The example code is here:
import liionpack as lp
import pybamm
import numpy as np
I_mag = 5.0
OCV_init = 4.0  # used for intial guess
Ri_init = 5e-2  # used for intial guess
R_busbar = 1.5e-6  # very small to simulate evenly distributed currents
R_connection = 1e-2
Np = 4
Ns = 1
Nbatt = Np * Ns
netlist = lp.setup_circuit(
    Np=Np, Ns=Ns, Rb=R_busbar, Rc=R_connection, Ri=Ri_init, V=OCV_init, I=I_mag
)

experiment = pybamm.Experiment(
    ["Discharge at 5 A for 30 minutes", "Rest for 15 minutes"],
    period="10 seconds",
)

output_variables = [
    'X-averaged negative particle surface concentration [mol.m-3]',
    'X-averaged positive particle surface concentration [mol.m-3]',
    ]

chemistry = pybamm.parameter_sets.Chen2020
parameter_values = pybamm.ParameterValues(chemistry=chemistry)
socInit=np.array([0.2, 0.3, 0.4, 0.5])

c_n_max = parameter_values["Maximum concentration in negative electrode [mol.m-3]"]
c_p_max = parameter_values["Maximum concentration in positive electrode [mol.m-3]"]

neg_init=np.zeros(Np*Ns)
pos_init=np.zeros(Np*Ns)

for i in range(Np*Ns):
    x, y = pybamm.lithium_ion.get_initial_stoichiometries(socInit[i], parameter_values)
    neg_init[i], pos_init[i] = x * c_n_max, y * c_p_max

parameter_values.update({"Initial concentration in negative electrode [mol.m-3]": "[input]"})
parameter_values.update({"Initial concentration in positive electrode [mol.m-3]": "[input]"})
inputs = {"Initial concentration in negative electrode [mol.m-3]": neg_init,"Initial concentration in positive electrode [mol.m-3]": pos_init}

print (np.unique(neg_init))

output = lp.solve(
    netlist=netlist,
    parameter_values=parameter_values,
    experiment=experiment,
    inputs=inputs,output_variables=output_variables
)
lp.plot_output(output)
lp.show_plots()

The output for surface concentration of negative and positive particles for the cells should look different as the initial SOC of cells are different. However, output is same for all the cells.

We could change the logic in the solvers to not use the initial_soc argument if is None and change the default and that should allow you to customize in this way or we could develop the logic to accept an array of initial_soc

This issue should now be addressed by PR #103 which is merged. See https://github.com/pybamm-team/liionpack/blob/main/examples/different_initial_soc.py for example and let me know if we can close this issue

@ksnvikrant also that other branch is redundant now so can you delete it?


fix: improve killing of devimint

fixes devimint doesn't die correctly if setup fails.

What about https://github.com/fedimint/fedimint/pull/2853 ?

If we shutdown here we may not have enough time to do the killing of subprocesses

Closed in favour of #2853


[WIP] FlinkEngineConnFactory - add default ha_mode 'NONE' for manage ec

What is the purpose of the change
EngineConn-Core defines the the abstractions and interfaces of the EngineConn core functions.
The Engine Service in Linkis 0.x is refactored, EngineConn will handle the engine connection
and session management.
Related issues/PRs
Related issues: close #590 close #591
Related pr:#591
Brief change log

Define the core abstraction and interfaces of the EngineConn Factory;
Define the core abstraction and interfaces of Executor Manager.

Checklist

[x] I have read the Contributing Guidelines on pull requests.
[ ] I have explained the need for this PR and the problem it solves
[ ] I have explained the changes or the new features added to this PR
[ ] I have added tests corresponding to this change
[ ] I have updated the documentation to reflect this change
[ ] I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
[ ] If this is a code change: I have written unit tests to fully verify the new behavior.

This pr has not been active for a long time, I will turn it off first. If you need to, you can re-open it.


Creating ECS TaskSet via CF has not been implemented

Trying to run a cloudformation template that provisions ECS Service, TaskDefinition, and TaskSet has resulted in following errors:
1. NotImplementedError: The create_task_set action has not been implemented

The Service and TaskDefinition has been created successfully though.
The error makes sense because I am using moto version 1.3.14 that was released in Nov'2019.  It looks like create_task_set feature has been added as recent as two days ago.
Is there a way I could utilize this new feature or when is this going to be released?

Hi @pinkjosh, a new dev-release is created for every commit to master. You can see the full list here: https://pypi.org/project/moto/#history
The TaskSet feature should be part of moto >= 1.3.15.dev979

@bblommers , Thank you for your prompt response on this question.  So is it okay if we start using the devl release instead. I am asking this from the perspective of moto libraries best practices.

Yes, @pinkjosh, it's encouraged to use the dev-releases. There's no defined schedule for official releases, so this is the best way to ensure you have the latest bugfixes/features.

Thanks @bblommers  I just tried using following moto releases:
moto==1.3.15.dev979
moto==1.3.15.dev984

This time no errors were shown but a message thats basically said the same thing. My console output below:
No Moto CloudFormation support for AWS::ECS::TaskSet
python3.7/site-packages/moto/cloudformation/parsing.py:330: UserWarning: Tried to parse AWS::ECS::TaskSet but it's not supported by moto's CloudFormation implementation
  resource_type
No Moto CloudFormation support for AWS::ECS::PrimaryTaskSet
python3.7/site-packages/moto/cloudformation/parsing.py:330: UserWarning: Tried to parse AWS::ECS::PrimaryTaskSet but
it's not supported by moto's CloudFormation implementation

resource_type
No Moto CloudFormation support for AWS::ECS::TaskSet
No Moto CloudFormation support for AWS::ECS::PrimaryTaskSet

Thank you @bblommers
No worries.  I tried working with TaskSet through mocked ECS boto3 calls and it appears to be working fine. I will proceed further.  Many thanks for your quick help.


producer: Wrap ExecutionException in a RuntimeException to silence doclint

What it says on the box. This lint rule breaks after upgrading ducktape containers to openjdk-17.
See https://github.com/redpanda-data/redpanda/pull/15951

Tested locally and the producer builds fine this way.

I don't really understand the implications of this.

Functionally, none as far as I can tell. Just bending to fit build tooling. All of our tests still pass. Updated the PR description with some detail.


use SourceLink.Create.CommandLine instead of Embed

You should use source linking instead of embedding the source files. This saves 275 KB for the nupkg. It goes from 12317 KB to 12042 KB.

@ctaggart, ok, thanks for the update, that's good news. I still need to do some manual testing that the newly created package works well when debugging in Visual Studio.
I'd still be interested what are the advantages of sourceLink over embedding the sources (I think the nuget size impact is negligible)? - Pragmatically, changes are always risky so I want to make sure we'll gain something my making this change.

I think some end users, Microsoft, and NuGet.org would prefer the smaller size. Another advantages is traceability to the source code repository. It would be nice to be consistent when what everyone else is doing including https://github.com/google/protobuf/pull/4179.

Sounds fair, thanks for the explanation. I'll do the manual testing and once that's done I'll approve.

testing packages are here: https://grpc-testing.appspot.com/view/Artifacts/job/gRPC_build_packages/1814/platform=windows/

They look good. The error about 5 files did not pass may be ignored. The tool does not work with the native dll's in the nupkg. Extracting and testing the portable pdb files works just fine.
C:\Users\camer\cs\SourceLink\build [master ≡]> dotnet sourcelink test C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3.nupkg
sourcelink test passed: lib/net45/Grpc.Core.pdb
sourcelink test passed: lib/netstandard1.5/Grpc.Core.pdb
unable to read debug info: C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3.nupkg
5 files did not pass in C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3.nupkg
C:\Users\camer\cs\SourceLink\build [master ≡]> dotnet sourcelink test "C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3\lib\netstandard1.5\Grpc.Core.pdb"
sourcelink test passed: C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3\lib\netstandard1.5\Grpc.Core.pdb
C:\Users\camer\cs\SourceLink\build [master ≡]> dotnet sourcelink print-json "C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3\lib\netstandard1.5\Grpc.Core.pdb"
{"documents":{"c:\\jenkins\\workspace\\gRPC_build_packages\\platform\\windows\\*":"https://raw.githubusercontent.com/ctaggart/grpc/b6f737ee7d9236511ed0f738fa7b6e7e2778b651/*"}}
C:\Users\camer\cs\SourceLink\build [master ≡]> dotnet sourcelink test "C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3\lib\net45\Grpc.Core.pdb"
sourcelink test passed: C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3\lib\net45\Grpc.Core.pdb
C:\Users\camer\cs\SourceLink\build [master ≡]> dotnet sourcelink print-json "C:\Users\camer\Downloads\Grpc.Core.1.9.0-pre3\lib\net45\Grpc.Core.pdb"
{"documents":{"c:\\jenkins\\workspace\\gRPC_build_packages\\platform\\windows\\*":"https://raw.githubusercontent.com/ctaggart/grpc/b6f737ee7d9236511ed0f738fa7b6e7e2778b651/*"}}

I've tested the packages manually and I was able to step into the code (after VS requested permission to download the sources from github) - so everything seems to work fine.
v1.9.0 was released yesterday, so we didn't make it in time, so the change will be in v1.9.1 later  and will be upmerged to upstream/master.
Thanks for the contribution!
A question:
Is there a way in VS to show me the original file in browser on github while debugging ("Open this file on Github") - that would be very useful feature to have.


db mode

A mode using a db to reduce the utxo memory requirement

Please paste the link to code/commit once done, I'm very interested in what are you cooking there. :)

Oh, interesting. So you make the DB reusable between runs. Smart! The initial perf. seem too slow. I've spotted couple of perf. optimization ideas. Maybe something will help.


Improve performance of showing suggestions

From https://github.com/tadashi-aikawa/obsidian-various-complements-plugin/issues/30

About 500000words. (using https://raw.githubusercontent.com/dwyl/english-words/master/words.txt)

Ah... Against one's expectations, It is not working correctly to memoize 😱

@chenyukang
Probably, It will fast more than 100 times!! ✨

Hi, @chenyukang!
I have uploaded the latest development version to tadashi-aikawa/obsidian-plugin-dev-sandbox 🚀  I re-architected the structure and improved both performances (about 10 times faster ✨) and quality of suggestions.
If there are no problems, I will release it tomorrow, officially :)
Kind regards.

Hi, @chenyukang!
I have uploaded the latest development version to tadashi-aikawa/obsidian-plugin-dev-sandbox 🚀 I re-architected the structure and improved both performances (about 10 times faster ✨) and quality of suggestions.
If there are no problems, I will release it tomorrow, officially :)
Kind regards.

Thanks, I will have a try.

Oh, NICE！

I suggest you add a default English dictionary, and give users an option whether to enable it.
The word list from Google search seems OK.

@chenyukang
Thank you for your try! and I'm very happy to look at your screenshot! 😆

I suggest you add a default English dictionary, and give users an option whether to enable it.
The word list from Google search seems OK.

Nice suggestion 👍   does it mean https://github.com/first20hours/google-10000-english/blob/master/google-10000-english.txt ?

@chenyukang Thank you for your try! and I'm very happy to look at your screenshot! 😆

I suggest you add a default English dictionary, and give users an option whether to enable it.
The word list from Google search seems OK.

Nice suggestion 👍 does it mean https://github.com/first20hours/google-10000-english/blob/master/google-10000-english.txt ?

I think this one is better : https://github.com/first20hours/google-10000-english/blob/master/20k.txt

Thanks! I will consider using it.

I'll close this issue since it releases v3.0.0.


Add "*.txt" files in path for GH workflow.

Description
Added "*.txt" files as a path in tests.yml to trigger the GH workflow.
Fixes issue #72.
Affected Dependencies
None
How has this been tested?
None
Checklist

[x ] I have followed the Contribution Guidelines and Code of Conduct
[ ] I have commented my code following the OpenMined Styleguide
[ ] I have labeled this PR with the relevant Type labels
[ ] My changes are covered by tests

Is this to trigger the workflow whenever there are changes in the requirments?

I think this is great but maybe it's better to make the file more explicit:
i.e.
Have requirements_dev.txt / requirements.txt instead of **.txt.
Because I don't think we want to trigger the workflow if we change anything in the license, for instance.

@gmuraru what do you think about this?

@gmuraru  arturomf94 I have another doubt, Issue #76 states that we remove all dependencies from requirements.txt and put them to setup.cfg and to remove requirements.txt from GA workflows. Isn't that issue in contradiction to this one?

@gmuraru @arturomf94 I have another doubt, Issue #76 states that we remove all dependencies from requirements.txt and put them to setup.cfg and to remove requirements.txt from GA workflows. Isn't that issue in contradiction to this one?

That's a good point! 🤔 @gmuraru?

There is also another requirement file named requirements_dev.txt.
Ideally, we would leave only the setup.cfg needed for installing sympc and the requirements_dev.txt would be used to install the devs stuff.

@RaghavPrabhakar66 - I think, for now, let's add both files explicitly: requirements_dev.txt and requirements.txt :+1:

@gmuraru @arturomf94 I have both requirements.txt to GA path.

Thank you @RaghavPrabhakar66. This looks great IMO. Unfortunately, the tests are currently failing but I believe it has nothing to do with your changes. I think it's best if we try to wait until these problems are fixed anyway and we could merge after that.

Hm... That's really weird. I don't know what could be going on here. @gmuraru - any ideas?

Hm... That's really weird. I don't know what could be going on here. @gmuraru - any ideas?

I re-run the tests now to see if it something that happens only sometimes (we should open a ticket if that is the case)


API: Raise FileNotFoundError in read_csv

For a nonexistent file, raise the more specific FileNotFoundError for Python >= 3.3 in read_csv.
Closes #14086.

Current coverage is 85.26% (diff: 75.00%)

Merging #14116 into master will decrease coverage by <.01%

@@             master     #14116   diff @@
==========================================
  Files           139        139
  Lines         50502      50506     +4
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits          43063      43065     +2
- Misses         7439       7441     +2
  Partials          0          0

Powered by Codecov. Last update be61825...4cfb15b

Looks good to me. Would be nice to also look if other readers raise IOError currently.

@jorisvandenbossche : read_hdf is another one that could raise a better error. :smile:

I think this also exists for json / excel.

@jreback : They give different errors, at least when I tried locally.

@gfyoung can you elaborate?

@jreback : Sorry, wasn't clear.  read_excel raises FileNotFoundError already (it calls Python's open function to open the file).  With read_json, I get this:
>>> read_json('nonexistent.json')
...
ValueError: Unexpected character found when decoding 'null'

I suppose that could be improved but seems a little different from what I'm fixing here.

ok, yes this should be ok for all other things. can you quickly see if read_stat/read_sas are ok? (I think things that call io/common/_get_filepath_or_buffer are fine. read_csv/read_hdf just do special things.
ping if ok

Looks good to me!

@jreback : Travis is passing. Ready to merge if there are no other concerns.


Add option to not update views

If you have huge views, Notion may fail to add new rows.
Fix https://github.com/jamalex/notion-py/issues/111

It would be nice to have this fixed, given this is causing a problem for me

I had to fork the package, apply the PR locally and build the package myself but I can confirm it works. Thank you @jean for your contribution. I am thankful to @jamalex for his amazing work but I am a little worried about the amount of useful PRs sitting here...

It looks as if this updated was already merged in through some other route, so I'll close this one. Thanks!!


Dropdown sorting is messy

I also noticed that the drop down of keyboards (first dropdown) is "semi-sorted" alphabetically. I suspect whoever put it together started with an alphabetic but then other keys were added after that. It would be great if this was sorted -- I've scrolled up and down that list a couple of time now. Even better if I could just start typing at the top - sorry feature creep request!
Originally posted by @svenski in https://github.com/Ciantic/keymapceditor/issues/3#issuecomment-589635994

:+1:


Implement client version validation

Currently client can connect to non compatible service version and get undefined behavior. See  https://github.com/uber-go/cadence-client/issues/290 for an example.
Service has to reject client connections from non compatible clients.

duplicate of #511


User Ops Round III

Rework user ops:

User ops take either a const or path as an argument.
The this value of the parent is carried through when a user op is invoked.

FWIW, I tested out this branch at commit 6fb851c and confirmed it seems to address what's described in #4650. That is, since we're no longer using spread to pass this into the user op, now primitive values make it through ok.
$ zed -version
Version: v1.8.1-32-g6fb851c9

$ cat kind-in-op-4663.zed
op CallMe(): (
  yield {kind: kind(this),ThisWas: this}
)
CallMe()

$ echo '{foo: "bar"} "hello"' | zq -z -I kind-in-op-4663.zed -
{kind:"record",ThisWas:{foo:"bar"}}
{kind:"primitive",ThisWas:"hello"}

It looks like #4658 is addressed as well. I can now pass in a dotted reference to a nested field or use field referencing with indexing and then reference the field ok from inside the user op.
$ zq -version
Version: v1.8.1-32-g6fb851c9

$ cat yield-in-op-4663.zed
op CallMe(f): (
  yield f
)

$ echo '{foo: {bar: "baz"}}' | zq -Z -I yield-in-op-4663.zed 'CallMe(foo.bar)' -
"baz"

$ echo '{foo: {bar: "baz"}}' | zq -Z -I yield-in-op-4663.zed 'CallMe(this["foo"]["bar"])' -
"baz"


network model

Thank you for your work. May I ask which file is your main network model.

The file is our main network model.


Added Stacks-CLI support

In order to enable the Stacks-CLI to create repos from the stacks-infrastructure-aks stack, it needs to have the stackscli,.yml file which tells the CLI what to do with the repository.
Currently the CLI cannot scaffold this repository, this change will allow this to happen.

/azp run


BUG when testing results of MTEB retrieval

I was running mteb run -m McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-supervised  -t DBPedia ArguAna NFCorpus FiQA2018   --output_folder results_0  --device 0
But I got：
File "/mnt/ceph_home/zhongtianyun2023/miniconda3/envs/mteb_new/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
AttributeError: 'LlamaBiModel' object has no attribute 'rotary_emb'
Batches:   0%|                                                                             | 0/1 [00:07<?, ?it/s]
[W729 12:01:59.244093981 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

Hi @tianyumyum,
what version of transformers are you using? llm2vec supports transformers>=4.39.1,<=4.40.2

I encountered the same issue yesterday while using this model. It started working after I downgraded the transformers to version 4.40.2.

Thanks！
I can run it now～@vaibhavad @lst627

@vaibhavad  Hello！ I have created a new conda environment with all the packages as in this issue.

I didn't get an error when using the command line, but I got an error Bus error (core dumped) when using python files.
I just ran the same code as you listed above in this site
`import mteb
if name == "main":
model_name = "McGill-NLP/LLM2Vec-Sheared-LLaMA-mntp-supervised"
model = mteb.get_model(model_name)
tasks = mteb.get_tasks(tasks=["AmazonCounterfactualClassification"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder="results")`
And the bug will happen when running experiments/mteb_eval.py.

@vaibhavad  Hello！ I have created a new conda environment with all the packages as in this issue.

I didn't get an error when using the command line, but I got an error Bus error (core dumped) when using python files.
I just ran the same code as you listed above in this site
`import mteb
if name == "main":
model_name = "McGill-NLP/LLM2Vec-Sheared-LLaMA-mntp-supervised"
model = mteb.get_model(model_name)
tasks = mteb.get_tasks(tasks=["AmazonCounterfactualClassification"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder="results")`
And the bug will happen when running experiments/mteb_eval.py.


feat(tutorial): complete step 5

Closes #
{{short description}}
Changelog
New

{{new thing}}

Changed

{{change thing}}

Removed

{{removed thing}}

https://carbon-tutorial-charlotte8699.eu-gb.mybluemix.net/#/

Per instructions in the tutorial step 5, please include your mybluemix.net URL for your deployed app in your pull request description.

Updated!


[NSE-341] fix maven build failure

What changes were proposed in this pull request?
This pr fixed the build failure on v2 with -Phadoop-3.2
fixed: https://github.com/oap-project/native-sql-engine/issues/341
How was this patch tested?
locally tested

@zhouyuan submitted a pr here

@zhztheplayer please take a look on this
master fail to build if -Phadoop3.2
this seems introduced in https://github.com/oap-project/native-sql-engine/pull/321


Typo in exercise of Chapter 10

In this line: https://github.com/KastnerRG/pp4fpgas/blob/master/sorting.tex#L287:
When the merge() function is called, whare are the possible relationships between i1, i2, and i3?

Finally fixed. Thanks!


Ziped Files and Descriptors not usable for some workflows

Describe the bug
(Preface that I have only looked at WDL workflows)
Many workflows in dockstore come from repositories that have complex structures where the primary descriptor actually imports workflows in other parent directories.
For example:
workflows
  |- wgs
    | - main.wdl
tasks
  |- do-stuff.wdl
structs
  | - human
    | - sample.wdl

When dockstore imports these workflows it assumes that the PRIMARY_DESCRIPTOR is at the root of the file tree and strips any leading path away, resolving all imports against the primary descriptor. This works when all imported files are siblings o of the primary descriptor or in child directories. This completely FAILS when the imports are in a totally separate parent directory. Since they are resolved relative to the root primary file their path ends up being prepended by back references ../../ instead of keeping the leading paths between main.wdl and do-stuff.wdl
main.wdl
../../tasks/do-stuff.wdl
../../structs/human/sample.wdl

So what is the big deal??
The problem with this is that workflows which have this structure are NOT RUNNABLE when passing the PLAIN_WDL/descriptor to any workflow engine (ie miniwdl or cromwell) they will not be able to resolve the workflow.
Additionally when trying to download the workflow files as a zip, the dockstore server is unable to create the zip file likely due to the fact that it cannot create paths with back references as zip entries

To Reproduce
Steps to reproduce the behavior:
Workflow ID: '#workflow/github.com/broadinstitute/warp/WholeGenomeGermlineSingleSample'
Version ID: tw_GL-2036_create_rtools_docker
Type: WDL

Got to the Swagger uI and navigate to the TRS request to get the files
Try out the request
Insert the above Workflow ID in the workflow id input
Insert the above Version ID in the version id input
Set the type to WDL and the format to zip
Click execute

Expected behavior
The Zip file to be downloaded with all of the files for the workflow in it
Actual Behaviour
{
  "servlet": "jersey",
  "message": "Bad Request",
  "url": "/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/WDL/files",
  "status": "400"
}

To Reproduce
Steps to reproduce the behavior:
Workflow ID: '#workflow/github.com/broadinstitute/warp/WholeGenomeGermlineSingleSample'
Version ID: tw_GL-2036_create_rtools_docker
Type: WDL
Descriptor URL: 'https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor
1 . Try to run the workflow using miniwdl (without passing any inputs)
miniwdl run https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor

Expected behavior
The workflow to be parsed and attempted to run (but fail if you do not provide inputs)
Actual Behaviour
miniwdl run 'https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor'
(https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor Ln 31 Col 1) Failed to import ../../../../../../tasks/broad/UnmappedBamToAlignedBam.wdl
HTTP Error 404: Not Found

To Reproduce
Steps to reproduce the behavior:
Workflow ID: '#workflow/github.com/broadinstitute/warp/WholeGenomeGermlineSingleSample'
Version ID: tw_GL-2036_create_rtools_docker
Type: WDL
Descriptor URL: 'https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor

Start cromwell: java -jar cromwell.jar server
Try to run the workflow using cromwell (without passing any inputs)

curl -X POST http://localhost:8000/api/workflows/v1 -F workflowSource='https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor'

Expected behavior
The workflow to be parsed and attempted to run (but fail if you do not provide inputs)
Actual Behaviour
Cromwell is not able to parse the workflow imports
[2023-07-12 11:01:23,90] [info] WorkflowManagerActor: Workflow 9b777f37-fa53-490b-9478-fd4d96b9a92c failed (during MaterializingWorkflowDescriptorState): cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anon$1: Workflow input processing failed:
Failed to import '../../../../../../tasks/broad/UnmappedBamToAlignedBam.wdl' (reason 1 of 1): Failed to resolve '../../../../../../tasks/broad/UnmappedBamToAlignedBam.wdl' using resolver: 'http importer (relative to https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/)' (reason 1 of 1): Failed to download https://dockstore.org/api/ga4gh/trs/tasks/broad/UnmappedBamToAlignedBam.wdl (reason 1 of 1): {"code":404,"message":"HTTP 404 Not Found"}
Failed to import '../../../../../../tasks/broad/AggregatedBamQC.wdl' (reason 1 of 1): Failed to resolve '../../../../../../tasks/broad/AggregatedBamQC.wdl' using resolver: 'http importer (relative to https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/)' (reason 1 of 1): Failed to download https://dockstore.org/api/ga4gh/trs/tasks/broad/AggregatedBamQC.wdl (reason 1 of 1): {"code":404,"message":"HTTP 404 Not Found"}
Failed to import '../../../../../../tasks/broad/Qc.wdl' (reason 1 of 1): Failed to resolve '../../../../../../tasks/broad/Qc.wdl' using resolver: 'http importer (relative to https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/)' (reason 1 of 1): Failed to download https://dockstore.org/api/ga4gh/trs/tasks/broad/Qc.wdl (reason 1 of 1): {"code":404,"message":"HTTP 404 Not Found"}
Failed to import '../../../../../../tasks/broad/BamToCram.wdl' (reason 1 of 1): Failed to resolve '../../../../../../tasks/broad/BamToCram.wdl' using resolver: 'http importer (relative to https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/)' (reason 1 of 1): Failed to download https://dockstore.org/api/ga4gh/trs/tasks/broad/BamToCram.wdl (reason 1 of 1): {"code":404,"message":"HTTP 404 Not Found"}
Failed to import '../../../../../../tasks/broad/Utilities.wdl' (reason 1 of 1): Failed to resolve '../../../../../../tasks/broad/Utilities.wdl' using resolver: 'http importer (relative to https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/)' (reason 1 of 1): Failed to download https://dockstore.org/api/ga4gh/trs/tasks/broad/Utilities.wdl (reason 1 of 1): {"code":404,"message":"HTTP 404 Not Found"}
Failed to import '../../../../../../pipelines/broad/dna_seq/germline/variant_calling/VariantCalling.wdl' (reason 1 of 1): Failed to resolve '../../../../../../pipelines/broad/dna_seq/germline/variant_calling/VariantCalling.wdl' using resolver: 'http importer (relative to https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/)' (reason 1 of 1): Failed to download https://dockstore.org/api/ga4gh/trs/pipelines/broad/dna_seq/germline/variant_calling/VariantCalling.wdl (reason 1 of 1): {"code":404,"message":"HTTP 404 Not Found"}
Failed to import '../../../../../../structs/dna_seq/DNASeqStructs.wdl' (reason 1 of 1): Failed to resolve '../../../../../../structs/dna_seq/DNASeqStructs.wdl' using resolver: 'http importer (relative to https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/)' (reason 1 of 1): Failed to download https://dockstore.org/api/ga4gh/trs/structs/dna_seq/DNASeqStructs.wdl (reason 1 of 1): {"code":404,"message":"HTTP 404 Not Found"}
	at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.cromwell$engine$workflow$lifecycle$materialization$MaterializeWorkflowDescriptorActor$$workflowInitializationFailed(MaterializeWorkflowDescriptorActor.scala:257)

┆Issue is synchronized with this Jira Story
┆Fix Versions: Dockstore 1.14.x
┆Issue Number: DOCK-2430
┆Sprint: 118 - Quillayute
┆Issue Type: Story

See 5182

Some thoughts from DMs

Probably affects CWL too, but not Nextflow (think it assumes the config is in the root)
We were kinda ambiguous on the format of the zip file. We could return a zip file starting at the "lowest" point of the data structure. We'd still need to return a header or somehow annotate/metadata the zip file to indicate which wdl was the primary descriptor. (or just correlate the zip and the json return)

The only way I can think of to fix resolving these types of imports against the /decriptor endpoint in TRS is to either:
Assume the client understands how to treat these strings in a special way, different then normal URL resolution semantics
require files to either be siblings or children of the primary descriptor

Looking specifically at the zip issue

Additionally when trying to download the workflow files as a zip, the dockstore server is unable to create the zip file likely due to the fact that it cannot create paths with back references as zip entries

This looks like a separate issue, basically, you need to make sure the media type matches the format or only use the media type
Or in curl terms
Right
curl -X 'GET' \
  'https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/WDL/files?format=zip' \
  -H 'accept: application/zip'

curl -X 'GET' \
  'https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/WDL/files' \
  -H 'accept: application/zip'

Wrong (gets 400)
curl -X 'GET' \
  'https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/WDL/files?format=zip' \
  -H 'accept: application/json'

Alternatively, the edit to the instructions is

Go to the Swagger uI and navigate to the TRS request to get the files
Try out the request
Insert the above Workflow ID in the workflow id input
Insert the above Version ID in the version id input
Set the type to WDL and the format to zip
Change the media type to application/zip
Click execute

or

Got to the Swagger uI and navigate to the TRS request to get the files
Try out the request
Insert the above Workflow ID in the workflow id input
Insert the above Version ID in the version id input
Set the type to WDL
Change the media type to application/zip
Click execute

Interestingly

Actual Behaviour
miniwdl run 'https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor'
(https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor Ln 31 Col 1) Failed to import ../../../../../../tasks/broad/UnmappedBamToAlignedBam.wdl
HTTP Error 404: Not Found

https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/WDL/descriptor/..%2F..%2F..%2F..%2F..%2F..%2Ftasks%2Fbroad%2FUnmappedBamToAlignedBam.wdl is a live link
Thinking miniwdl must be collapsing the ../ parts

Tinkering with miniwdl
$ miniwdl run https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor
before:https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor../../../../../../tasks/broad/UnmappedBamToAlignedBam.wdl
after:https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor/..%2F..%2F..%2F..%2F..%2F..%2Ftasks%2Fbroad%2FUnmappedBamToAlignedBam.wdl
before:https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor/..%2F..%2F..%2F..%2F..%2F..%2Ftasks%2Fbroad%2FUnmappedBamToAlignedBam.wdl../../tasks/broad/Alignment.wdl
after:https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor/..%2F..%2F..%2F..%2F..%2F..%2Ftasks%2Fbroad%2FUnmappedBamToAlignedBam.wdl/..%2F..%2Ftasks%2Fbroad%2FAlignment.wdl
(https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor Ln 31 Col 1) Failed to import ../../../../../../tasks/broad/UnmappedBamToAlignedBa
![](https://media.tenor.com/Ndd63HyG3KcAAAAd/julia-louis-dreyfus-nervous-laughter.gif)
m.wdl
(../../../../../../tasks/broad/UnmappedBamToAlignedBam.wdl Ln 19 Col 1) Failed to import ../../tasks/broad/Alignment.wdl
HTTP Error 404: Not Found : version found, but file not found (bad filename, invalid file, etc.)

Particularly
https://dockstore.org/api/ga4gh/trs/v2/tools/%23workflow%2Fgithub.com%2Fbroadinstitute%2Fwarp%2FWholeGenomeGermlineSingleSample/versions/tw_GL-2036_create_rtools_docker/PLAIN_WDL/descriptor/..%2F..%2F..%2F..%2F..%2F..%2Ftasks%2Fbroad%2FUnmappedBamToAlignedBam.wdl/..%2F..%2Ftasks%2Fbroad%2FAlignment.wdl

Uh oh


docs: extended contributing requirements

Discovered in #1358

@aryamohanan @abhilash-sivan Please execute locally & approve.

See https://github.com/instana/nodejs/pull/1358/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519R43

I could see a contradictory statement here https://github.com/instana/nodejs/pull/1378/files#diff-eca12c0a30e25b4b46522ebf89465a03ba72a03f540796c979137931d8f92055R127

I could see a contradictory statement here https://github.com/instana/nodejs/pull/1378/files#diff-eca12c0a30e25b4b46522ebf89465a03ba72a03f540796c979137931d8f92055R127

Good catch 👍
Fixed https://github.com/instana/nodejs/commit/dcc4b744ad1a4a538bf20d05b20fedf301d99e05


Add support for InitialWorkDirRequirement

In CWL InitialWorkDirRequirement can be used at the start of a commandline tool to specify files that must also be placed in tmp folder during processing. In nextflow this would involve just adding the additional files as inputs I believe.

Yes exactly, something like:
process foo {
  input:
   file 'x' from file('x')
   file 'y' from file('y')
}

Yes it required a very minimal adjustment to get the CWL_RNAtoy converting and running correctly. I need to create a more complex test case. This will also require javascript as the InitialWorkDirRequirement can be an expression.

The InitialWorkDirRequirement is currently supported. The expression portion will be fixed with continued work on JS handling.


Upgrade ahash to 0.8

Which issue does this PR close?

Closes https://github.com/apache/arrow-rs/pull/2395
Rationale for this change
Updating dependency, gets some fixes and a small perf improvement.

What changes are included in this PR?

Are there any user-facing changes?

 compile_error!("the wasm32-unknown-unknown target is not supported by \
236 | |                         default, you may need to enable the \"js\" feature. \
237 | |                         For more information see: \
238 | |                         https://docs.rs/getrandom/#webassembly-support");

wasm build is failing

 compile_error!("the wasm32-unknown-unknown target is not supported by \
236 | |                         default, you may need to enable the \"js\" feature. \
237 | |                         For more information see: \
238 | |                         https://docs.rs/getrandom/#webassembly-support");

wasm build is failing

Thanks, yeah I noticed. Let's see how we can fix this.

@tustvold switching to a compile-time rng for wasm helps.


Shopify::Base.connection.response nil for private apps

$ cat orders.rb

require 'shopify_api'
require 'pry'

def initialize
  shop_url =<EMAIL_ADDRESS>  ShopifyAPI::Base.site = shop_url
  ShopifyAPI::Base.api_version = '2020-01'
end

initialize
orders = ShopifyAPI::Order.find(:all, params: { limit: 35, stats: :any })

$ bundle exec ruby orders.rb
Traceback (most recent call last):
	6: from orders.rb:12:in `<main>'
	5: from /usr/local/var/rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/activeresource-5.1.0/lib/active_resource/base.rb:974:in `find'
	4: from /usr/local/var/rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/activeresource-5.1.0/lib/active_resource/base.rb:1073:in `find_every'
	3: from /usr/local/var/rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/activeresource-5.1.0/lib/active_resource/base.rb:1101:in `instantiate_collection'
	2: from /usr/local/var/rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/activeresource-5.1.0/lib/active_resource/base.rb:1101:in `new'
	1: from /usr/local/var/rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/shopify_api-9.0.0/lib/shopify_api/paginated_collection.rb:7:in `initialize'
/usr/local/var/rbenv/versions/2.6.0/lib/ruby/gems/2.6.0/gems/shopify_api-9.0.0/lib/shopify_api/paginated_collection.rb:52:in `pagination_link_headers': undefined method `[]' for nil:NilClass (NoMethodError)

Digging through the code it seems that the response is always nil.  The response body however does contain the expected orders, it's just breaking on pagination.
Even stranger, no issues when using the same install on a public app ShopifyAPI::Order.find(:all, params: { limit: 35, stats: :any }).
So requests are being made, auth is good, results are present, just that Shopify::Base.connection.response is nil and barfs trying to get Shopify::Base.connection.response["Link"].
It does look like ResponseCapture is being prepended properly and handle_response is being called (I'm actually not 100% sure if this is responsible for setting the response but thought I'd include it anyhow).
[1] pry(#<ShopifyAPI::Connection>)> @response = super(ShopifyAPI::MessageEnricher.new(response))
=> #<Net::HTTPOK 200 OK readbody=true>

Let me know if I can provide more details.


[feature request] Hide textbox

Hi,
it would be nice to hide the Textbox, as I only need to select a color but the user does not need to know the text value (hex string).
Could you make a xaml-setable property like IsHexVisible or something like this?
Thank you very much in advance.

Hello!
For consistency, I added the property ColorPickerViewModel.IsHexEnabled. The NuGet package should be available soon 😎

https://github.com/daltonks/Spillman.Xamarin.Forms.ColorPicker/commit/645b7eae929f44259ce840e5d48376b9fa3397be

Awesome thank you very much for the quick response and solution :)


Xamarin support?

Is there a Xamarin port for the library?

I’m not familiar enough with Xamarin, but the process of creating a binding for a native iOS library is documented in the Binding iOS Libraries guide. Since XCDYouTubeKit can be built as a framework and as a pod, you should easily get it working.


Does this support sharded Mongo Databases?

I haven't tested it, but it should work.
The primary key (_id) is the cache key and is the only candidate for the shard key. Also, all the operational requirements should be met.
Please, tell me if you are able to test it, otherwise I'll try to get an access to a sharded cluster and do some testing.

In case of a replica set you should pay attention to the read preference mode, which should be primary (the default).
Let me know how it goes :-)

Turns out this works perfectly fine. Some other piece of the application was causing it to have an invalid value.
Logging FTW!


Fix broken package substitution in starter archetype

Fixes a small substitution error introduced into the starter archetype.
Introduced here: https://github.com/apache/beam/commit/248b79452cd49e1186aa93177988a08169ac802e#diff-51014d28208054c58ec4aa4fa3e49553ac68fb80355511b86e813be20cd850d8

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

@robinyqiu - not sure exactly who to ask for a review but thought I should notify you as the reviewer on the PR where this was introduced

Please find a proper reviewer, @aaltay Thanks!

R: @tysonjh

Hm... The precommit is failing on the spotbugs check:
19:47:04 Execution failed for task ':sdks:java:harness:spotbugsMain'.
19:47:04 > A failure occurred while executing com.github.spotbugs.snom.internal.SpotBugsRunnerForWorker$SpotBugsExecutor

Hm... The precommit is failing on the spotbugs check:
19:47:04 Execution failed for task ':sdks:java:harness:spotbugsMain'.
19:47:04 > A failure occurred while executing com.github.spotbugs.snom.internal.SpotBugsRunnerForWorker$SpotBugsExecutor

Spotbugs is hitting a false positive here, the call is clearly synchronized, so we can go ahead with this change:
In class org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessor
Field org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessor.instructionId
Synchronized 66% of the time
Unsynchronized access at ProcessBundleHandler.java:[line 752]
Synchronized access at ProcessBundleHandler.java:[line 748]
Synchronized access at ProcessBundleHandler.java:[line 744]

Run Java PreCommit

Run Java PreCommit

Precommit failures seem unrelated.

Run Java PreCommit

Run Java PreCommit

@kennknowles can you take a look & merge? This LGTM.

Or @aaltay  (didn't see you here before commenting)


⚠ Samsung has degraded performance

In 86d2202, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2008 ms

Resolved: Samsung performance has improved in 51b56cd.


Zulip sends emails (tons of them) when there is an error

Hello,
I wasn't sure where to ask about that. I have a slight problem with my zulip deployment.
Zulip sends an email when it sees an error. I was wondering if there is a way to disable this behaviour.
I can see here a const that could be useful: https://github.com/zulip/zulip/blob/b3f18b805d2054a8e4e11f7ec363a7f2367c8c6c/zproject/default_settings.py#L519, or maybe the list of admins here: https://github.com/zulip/zulip/blob/b3f18b805d2054a8e4e11f7ec363a7f2367c8c6c/zproject/default_settings.py#L521. Maybe if I make these settings empty?
It's probably a weird use case, but I have zulip deployed on a kubernetes cluster. Somehow the connection with rabbitmq got interrupted, and the zulip container seems to have been killed. Except that k8s respawned the container, again and again, and I received ~400 times the same email haha

To stop Zulip from sending error emails, you can try setting ADMINS and SERVER_EMAIL to None or empty values in your settings.py file, which overrides default_settings.py. Here’s what you can do:
Override Settings:
Open zulip/settings.py (or another settings file you're using for custom configurations).
Set ADMINS = () and SERVER_EMAIL = "".
Suppressing Error Emails:
For added control over error reporting, you can adjust Django's logging settings to disable email notifications specifically for error logs:
LOGGING['loggers']['django.request'] = {
'handlers': ['console'],
'level': 'ERROR',
'propagate': False,
}
Customizing on Kubernetes:
If you're using a config file specific to Kubernetes, apply these changes in that config or via environment variables if the settings are set dynamically.
This should suppress those error emails without affecting other logging and monitoring in Zulip. If issues persist, check if there’s another source triggering emails (like a cron job or another monitoring script)

So I looked into the zulip pod and got the content of settings.py. Surprisingly, ADMINS and SERVER_EMAIL aren't there. Is that expected? I can add them manually of course, or through the env of my kubernetes deployment (trying that at the moment), but I'm just curious.

Yes, that's expected -- those settings default to ZULIP_ADMINISTRATOR if not set explicitly, as shown in the code above.
https://github.com/zulip/docker-zulip/issues/426 has some conversation that may be helpful for debugging how the connection got killed, BTW -- some container runtimes kill "idle" TCP connections.
Error emails are rate-limited to 1 per traceback per 10 minutes, using redis as storage for the rate limiting state, with fallback to in-process memory (which wouldn't work here given the server being restarted).
Did the emails have different tracebacks? I'm wondering if the issue here is that ZulipLimiter, the filter we have on top of Django's built-in error emails feature, needs a global limit as well, or if the code wasn't even running.

Yep the traceback was the same all the time. The connection to redis got killed because, well, I inadvertently unplugged the power brick of the node where redis is running :) (I force redis to be on a specific node).
Anyway, I set ADMINS in my config and I haven't seen any other email coming (I tried unplugging the node again haha), so I think it worked. Thanks for the help


Updated Project Status and Roadmap

I have reached out via Twitter and email several times over the last 6 months to @michenriksen with no response. I am looking to gauge the interest of those that use this with helping to maintain it. I have been using this for about 2 years now and forked it long ago for a separate project adding support for Github Enterprise, Gitlab CE, and Enterprise as well. I have some very rough code for using this with Perforce also.
My goal would be to bring some additional modularity to this. I would fork and then merge the existing PR's that make sense and then start to work through any issues remaining in this repo and look to transfer the rest to the fork. I welcome anyone who wants to help with the triage process.
Once that is done and stability and usability have been ensured I would like to explore using plugins for various types of source control systems such as perforce, svn, mercurial, as well as sites like Pastebin. Again, any help is welcome.
Another improvement I started to make was the move away from regex's or at least make them a secondary discovery method in favor of language-specific parsers. Again my thought would be to create a pluggable architecture that would allow others to contribute without having to learn the entire codebase. The first step in this though would be to break out the regex's into a separate file or repo and write tests for them and get some baseline metrics on their performance.
This is a lot of work but I have a vested interest in expanding and updating the codebase and would love to preserve the community and experience others have built up over the years and continue with this project using as much of the great code and effort already put into this from so many people.

I am using this ]fork](https://github.com/mattyjones/gitrob) and have merged in the following pull requests:

[X] 140 @josephmilla
[ ] 141 @josephmilla
[X] 149 @betobrandao
[X] 153 @micksmix
[X] 155 @dcepler
[X] 156 @dcepler
[X] 158 @dcepler
[X] 159 @muffix
[ ] 164 @wdahlenburg
[X] 177 @plasticuproject
[X] 188 @da-edra
[ ] 194 @codeEmitter
[ ] 199 @HollywoodMarks

I am happy to take additional pull requests in that fork to continue moving this project forward.
I am currently taking an inventory of the issues here and looking at a plan forward for any remaining ones.

I am starting to triage issues now and will track the list of current issues here. If possible please open all new issues in this fork. Thanks!

[ ] #197 @spekulatius
[ ] #196 @harsh-kk
[ ] #195 @shadihh9
[ ] #193 @codeEmitter
[ ] #192 @intruderketo
[ ] #191 @kirtiso
[ ] #190 @dzxindex
[ ] #189 @melardev
[ ] #187 @da-edra
[ ] #186 @ranjithprethan
[ ] #185 @dzhenway
[ ] #184 @prosecurity
[ ] #183 @SaurabhDev
[ ] #182 @SaurabhDev
[ ] #181 @newpIDC
[ ] #180 @jarvis-parker
[ ] #179 @soupnatzi
[ ] #178 @iamwilhelm
[ ] #176 @VikzSharma
[ ] #175 @leroywomack
[ ] #172 @mixua
[ ] #171 @igorljubuncic
[ ] #170 @SeanAmmirati
[ ] #169 @farmankhan977
[ ] #167 @TheHmadQureshi
[ ] #166 @derricksong
[ ] #165 @wdahlenburg
[ ] #163 @wdahlenburg
[ ] #161 @payloadartist
[ ] #157 @dcepler
[ ] #154 @dcepler
[ ] #152 @micksmix
[ ] #151 @satssin
[ ] #150 @ajholland
[ ] #148 @nuritizra
[ ] #147 @sohailnajar
[ ] #146 @hanjian007
[ ] #145 @0xtavian
[ ] #144 @josephmilla
[ ] #143 @menzow
[ ] #142 @josephmilla
[ ] #139 @josephmilla
[ ] #138 @josephmilla
[ ] #133 @bugbaba
[ ] #132 @skim-milk
[ ] #131 @jaikishantulswani
[ ] #130 @ScatteredThoughts

I am now closing out this issue and tracking everything in the new fork. All pull requests with the exception of one have been merged. All open issues have been triaged and either moved or closed. One door shuts, and a new one opens.

Hey thanks for reaching out.  I’ve recently been in contact with the original maintainer through a friend. Would you be willing to give him a little more time to look things over please?
Given the nature of the refactorings I needed to do to add the features I did, my PR will most likely not merge cleanly on your fork.  It was the perfect stepping off point for what we needed to accomplish for one of operations.  Red Team at GitLab is still actively working on enhancements and fixes and there will be lots more work coming.  I’m more than willing to share these contributions with the community and would be happy to work with you and the original maintainer if interested.
For context, in order for our team to track efforts more easily given the recently stale status here, we’ve created a pull mirror of this repo to track issues temporarily:  https://gitlab.com/gitlab-com/gl-security/gl-redteam/gitrob/-/issues
I have some ideas on options to move forward with project maintenance we could discuss if you’d like.

@codeEmitter I am happy to sync with you and @michenriksen as well as figure out a path forward for all of this. If your code has diverged I am happy to roll my sleeves and help to get it cleanly merged as GitLab support was something I didn't even implement.
My reasoning for hard forking this was along the same lines as yours. I built a tool based on this that I am not able to share in its current form. It was a more modular version with the following functionality

Github support
Github Enterprise support
more extensible and modular (using cobra)
10x speed improvement (think masscan) through continued use of threads
full support for telemetry and metrics to aide in tuning the performance
full logging (using logrus)
unit/integration tests
ability to scan a single repo
ability to scan offline git repos
ability to be run as a service in a CI/CD pipeline
ability to run in AWS Lamdba at-will when necessary
all sig's are broken out into a separate repo and independently versioned and releasable
the ability to display the secret itself
support for sqlite giving the ability to just do a delta scan or pause/resume
ability to set the commit depth of a scan
ability to read configs from a yaml file (using viper)

My version was designed for extensibility and brute, raw performance. I pulled the plug on my dev work about a year ago but the final thing I was working on was a grpc/rpc implementation to allow the addition of new target capabilities or detection methods without having to know the entire codebase. Go plugin was not yet available and it does not work on Windows. When it comes to performance, scanning 100k offline repos with a combined size of multiple TB to full depth in 8-9 hours was entirely possible. I have the release to bring these features into the open but have to implement them cleanly.
There is a fair amount of work here and the tool would look differently by the time I was done so before I started I was going to create a requirements doc like any other project and solicit feedback from the community to see how useful these features would be to the greater world and not just my specific use cases.

@codeEmitter @michenriksen Now that I had some non-keyboard time to think about this, here are my current thoughts on the matter. Full disclosure there several people on my team that also use our fork of gitrob and miss the features we implemented and want to recreate them. We tried to get our employer to maintain my fork officially but they gave it a pass so I, being the lead dev, got the ok to have at it. A few people on the team may contribute as time goes on though.
@michenriksen If you want to resume day to day managing of the project again that is fine by me and I will return to the backseat and try to pull in my fork of all the pull requests and bug fixes. I understand how much work can be involved as this is not my first or only rodeo and nowhere near the biggest thing I maintain or advise on. With that said if you have other commitments and simply don't have the time or want to invest the time but don't want to see the project languish or you don't want to give it up I can understand that as well. In that case, if you would like, we can continue with the hard fork, which means I have not forked the git commit history, I pulled the code and removed .git/ and did a clean git init You can close out these issues and pull requests, archive the repo, and my fork can become the new codebase.  That is not how I work. I grew up on BBS's and USENET when open source was still just chatter at late night MIT hackfests. I would be thrilled to give you full admin rights to my repo and we can collaborate together on the various lines of thinking for moving this forward.
As I said above, documentation is a huge frustration factor and there were several issues that either requested demos/docs or could have been solved with more detailed docs. At some point sooner rather than later I was going to grab the DNS for gitrob.io and standup a quick Hugo site with Github pages and throw all the docs, release info, a demo or two and maybe some related posts on secrets management, and the failing of it as it relates to AWS S3, GDrive, Github/GitLab, Perforce, etc.
With all of that said I still feel this tool is the ideal mix of blue team and red team methodologies. I have used it for both, in a CIO/CD pipeline for catching things before they get built and also for sifting through a company's public repos when I have the need to. Please feel to continue this conversation in this thread, or if you need to reach me privately, you are welcome to DM me on Twitter. I am going to continue on with my for and cleanup though as that is the best path forward at the moment.

@codeEmitter I took a look at the repo you linked me and your team has some damn good code and ideas in there. A lot of it is not where I see my needs focused but the features and direction are great! I honestly don't think it would be hard to do a transplant if we sat down over time and looked at it from an engineering standpoint and carved out the code.
One of the big things I am looking at is breaking this out into a few packages, and if/when that happens it would be a great time to then take a frozen branch of your code and start to slowly implement it package by package, such as front end, GitLab targets, new regex's and discovery methods, etc. At least from the mile-high view that is what I am looking at. Once that is done we can look at pulling in smaller chunks and then if you wish maintaining parity where you could simply pull from upstream when you wanted or open a new request.
As I have done in the past with major contributors or larger entities, I would be happy to at some point give you commit bit to the repo and the ability to review code to make it easier to function as a cohesive unit. @michenriksen if we go the route of keeping my fork I would be more than happy to do the same and give you admin rights as this was your baby for a long time and you deserve to bask in the glory. My reason for doing what I am doing is that I believe in community and shared responsibility first so I tend to fight to not splinter a repo or community unless absolutely necessary.
I kinda think going with my fork is a little easier as I have open source project management experience as well as developer advocate experience. The repo I have is already plugged in with issue tracking and release management via Zenhub, automated tests via CircleCI, code coverage, vulnerability scanning with synk, and code quality via code climate. These tools make it much easier to maintain and manage a codebase from a birds-eye view, especially when they are all integrated together.

It is still very much a beta project and there are some know bugs with the web interface that will be fixed shortly but wraith is functional and stable at last. The gitlab redteam had a fork of gitrob that they added features to and fixed bugs on. I also had a fork of this from an internal project at my work. I, in conjunction with @codeEmitter combined both projects into a new project, wraith, and have been adding features and fixing bugs there.
There are gaps in the documentation and a few things were broken in the brain surgery, we also uncovered several nasty bugs affecting the functionality of gitrob that we are still working to fix. It is fairly stable though and will work, just not to its full potential yet.
Pull requests and bug/feature requests are welcome!


I met some error about subplot1 in line 97 in main function

Hello, do you have the library for subplot1?

sorry, I don't have the library for subplot1

Check this link out, I don't have code on my hand currently but you can google out some candidates of function subplot1.

thanks a lot，I will try it
---Original---
From: "Jun @.>
Date: Tue, May 24, 2022 11:11 AM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [HybridRobotics/Quad-Load-Pulley] I met some error about subplot1in line 97 in main function (Issue #2)
Check this link out, I don't have code on my hand currently but you can google out some candidates of function subplot1.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Feel free to close this issue if it's resolved.

ok
---Original---
From: "Jun @.>
Date: Sun, May 29, 2022 13:14 PM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [HybridRobotics/Quad-Load-Pulley] Error about subplot1 in line 97 in main function (Issue #2)
Feel free to close this issue if it's resolved.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

quite thanks

surely not....
---Original---
From: "Jun @.>
Date: Tue, May 24, 2022 10:57 AM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [HybridRobotics/Quad-Load-Pulley] I met some error about subplot1in line 97 in main function (Issue #2)
Hello, do you have the library for subplot1?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

^ Please don't spam in the discussion of the issue. Thanks.


Invalid file path 'E:\BuildAgents\agent-1_work\58\s\CommonScripts\UpdateVersion.ps1,'. A path to a .ps1 file is required.

Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: here
Enter Task Name: here
list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment

Server - Azure Pipelines or TFS on-premises?

If using TFS on-premises, provide the version:

If using Azure Pipelines, provide the account name, team project name, build definition name/build number:

Agent - Hosted or Private:

If using Hosted agent, provide agent queue name:

If using private agent, provide the OS of the machine running the agent and the agent version:

Issue Description
[Include task name(s), screenshots and any other relevant details]
Task logs
[Enable debug logging and please provide the zip file containing all the logs for a speedy resolution]
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs
[Insert error from the logs here for a quick overview]

Hi @farazahmed879 Please provide more details about your issues in the description

Hi @farazahmed879 Is it still actual for you?

Hi @farazahmed879 closing this one because of lack of details - please let us know if it's still actual for you.


fuzzing broken in CI

The fuzzing task that has been notoriously hard to maintain is once again broken. It believes one of the original inputs causes an immediate crash in Rumur, and thus incorrectly exits early. I can't reproduce this locally, so presumably it's an OOM or something similar. The simplest way to fix this may be migrating this task off Travis.

The fuzzing job was removed in bfefe96b7e7d46922bfa12b1960d1efe266d2a8c.


New function suggestions

Hi,
I'd like to request the addition of the following functions:
Entity.isImmobile(entity)

Block.isFlammable(id)
Block.setFlammable(id)
Block.getExplosionResistance(id)
Block.getLightLevel(id,data)
Block.getLightOpacity(id,data)

Item.setStackSize(id,data,stackSize)
Item.getStackSize(id,data)
Item.getCategory(id,data)
Item.overrideDefaultName(id,data,newName)
Item.getStackedByData(id)

Thanks!

You can already do almost all of these with existing functions.

I'm after the missing getters for the existing setters.

I do like the get / set stack size though...that would help although you can get a rough idea by looking through the list of entities and checking their coordinates.  Anything less than 0.5block away can be added together as a stack...again, close, but not what you are asking.

yeah, I'm not saying what you are asking for is bad, just that you can work around most of them for now - so don't let it hold your development up!  :-)

@zedman2000 actually I meant that as in, get the max amount of that item you can have in 1 inventory slot, like 16 for signs, and the ability to change this

@zedman2000 yeah I gotcha

@dktapps understood.  Maybe add a 1 sentence under the listed function names so we understand what you are after.  You can see how easily they can be misinterpreted.  Generally speaking, I will just code the limit into my mod using the wiki.  Regardless, if you just set a stack of signs to be any number up to 255, it will be set...you just can't use the game mechanic to do it as it will respect the "16" for signs.  Maybe that will help you work around for now?

@zedman2000 yeah I guess. It'd be a little awkward but it would work :)


keyboardType issue

How can I fix below issues
1: returnKeyType="next" always work as return when I use keyboardType="number"
2: noDecimals="true" not working

@diadal What's noDecimals? Perhaps you're confusing this with nativescript-numeric-keyboard?


ISPN-10509 updating the doc index page

https://issues.jboss.org/browse/ISPN-10509

Looks good, could you squash the two commits?

Integrated, thanks @oraNod !


Add the 3d scatter selection toolbar

I divided the scatter (and later volume) toolbar from the Base Toolbar, which enables us to implement some advanced selections based on different data sets in the future.
Two problems to be solved within this PR:

Subset doesn't work well, only one point will be highlighted during the selection.
Two toolbars are shown in the same window (as see below), I guess it's because I use super().__init__ in the ScatterSelectionToolbar so the __init__() of Base Toolbar is called twice.

@PennyQ - thanks for your work on this! I opened a PR to your branch to fix the issue of the toolbar appearing twice: https://github.com/PennyQ/glue-3d-viewer/pull/1 - can you merge that PR?

When I try the selections, they do seem to work (although at the moment the selections don't propagate to other viewers) - can you clarify what you mean by only one point will be highlighted?
Before we can merge this, I think the following need to be done:

[ ] Merge https://github.com/PennyQ/glue-3d-viewer/pull/1 to fix the double toolbar
[ ] Implement the selection using the subset state infrastructure, and make sure:

[ ] Selections propagate to other viewers
[ ] The and/or/replace/etc modes work correctly

[ ] Show the outline of the selections (e.g. circle, rectangle, etc.) while drawing them

@astrofrog Oh yeah, it has been fixed and the 'simple' point selection works in the scatter plot now :)

This is fantastic - merging! :)


NETOBSERV-92 : Fixing tests coverage and adding coverage-report targets

Without -coverpkg=./... option, code coverage is only done for the package beeing tested.
Before PR:

total:														(statements)			41.5%

After:

total:															(statements)			71.6%

Could someone with Goland or Intellij check we have now the same coverage ?
May be the fix here is not enough.

@OlivierCazade these are the reports for my Intellij IDEA run in master:

Strangely, my reported total coverage is now 41.5 🤦🏻 so I can't verify your numbers, but definitely they look much better and more consistent with the numbers I was getting some weeks ago.

Setting the coverpkg argument in intelliJ idea reports me more realistic coverage data:

Thanks @mariomac  for the feedback.
I don't have the coverage per package but based on your screenshots, I have the same coverage for the following files :

main.go
flowcollector_controller.go
flowcollector_objects.go

/approve


Doesn't seem to work with Unity 2020.1

After dropping Editor.log into the window it does not show a tree.

It seems to work for me in 2020.3
Sometimes you have to clear the log and make sure it actually gets written to after a build.