[Bug] Galera complete cluster failure recovery

Yesterday evening I've deployed mariadb-operator for a development Galera cluster. I then started importing all table structures, which went fine. After that I tried to import some big sql data files in parallel over the course of the night. Biggest one being a 13G file. All files were originally created using mysqldump from a production database. Unfortunately when I woke up, I saw that all 3 of galera's instances had crashed and that the pods were in a CrashLoop.

Describe the bug

Unfortunately, this cluster does not have additional logging yet. So figuring out why the first crash happened is not an option anymore. What is an option, is to bootstrap the cluster and resume operatings from the last known good state. Unfortunately, mariadb-operator does not seem to provide a way to do this. All tries to start the servers end in the following logs:
2023-08-02 07:09:00+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
2023-08-02 07:09:00+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-08-02 07:09:00+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
2023-08-02 07:09:01+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2023-08-02  7:09:01 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 1
2023-08-02  7:09:01 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2023-08-02  7:09:01 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2023-08-02  7:09:01 0 [Note] WSREP: wsrep_load(): Galera 26.4.14(r06a0c285) by Codership Oy<EMAIL_ADDRESS>loaded successfully.
2023-08-02  7:09:01 0 [Note] WSREP: Initializing allowlist service v1
2023-08-02  7:09:01 0 [Note] WSREP: Initializing event service v1
2023-08-02  7:09:01 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2023-08-02  7:09:01 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 0
2023-08-02  7:09:01 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: 1fae98e4-30a7-11ee-b48b-1a696f898dba
Seqno: 4930 - 4963
Offset: 1280
Synced: 1
2023-08-02  7:09:01 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 1fae98e4-30a7-11ee-b48b-1a696f898dba, offset: 1280
2023-08-02  7:09:01 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
2023-08-02  7:09:01 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2023-08-02  7:09:01 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 4930-4963
2023-08-02  7:09:01 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...  0.0% (       0/56867952 bytes) complete.
2023-08-02  7:09:01 0 [Note] WSREP: Recovering GCache ring buffer: found 6/40 locked buffers
2023-08-02  7:09:01 0 [Note] WSREP: Recovering GCache ring buffer: free space: 79235968/134217728
2023-08-02  7:09:01 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (56867952/56867952 bytes) complete.
2023-08-02  7:09:01 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = mariadb-galera-dev-0.mariadb-galera-dev-internal.mariadb-operator.sal; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evsk_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gc= 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fgcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit =<PHONE_NUMBER>854775807;
2023-08-02  7:09:01 0 [Note] WSREP: Start replication
2023-08-02  7:09:01 0 [Note] WSREP: Connecting with bootstrap option: 0
2023-08-02  7:09:01 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2023-08-02  7:09:01 0 [Note] WSREP: Using CRC-32C for message checksums.
2023-08-02  7:09:01 0 [Note] WSREP: backend: asio
2023-08-02  7:09:01 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2023-08-02  7:09:01 0 [Note] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2023-08-02  7:09:01 0 [Note] WSREP: restore pc from disk failed
2023-08-02  7:09:01 0 [Note] WSREP: GMCast version 0
2023-08-02  7:09:01 0 [Note] WSREP: (750c83dc-803e, 'tcp://<IP_ADDRESS>:4567') listening at tcp://<IP_ADDRESS>:4567
2023-08-02  7:09:01 0 [Note] WSREP: (750c83dc-803e, 'tcp://<IP_ADDRESS>:4567') multicast: , ttl: 1
2023-08-02  7:09:01 0 [Note] WSREP: EVS version 1
2023-08-02  7:09:01 0 [Note] WSREP: gcomm: connecting to group 'mariadb-operator', peer 'mariadb-galera-dev-0.mariadb-galera-dev-internal.mariadb-operator.svc.clusterb-galera-dev-1.mariadb-galera-dev-internal.mariadb-operator.svc.cluster.local:,mariadb-galera-dev-2.mariadb-galera-dev-internal.mariadb-operator.svc.cluster.local:'
2023-08-02  7:09:01 0 [Note] WSREP: (750c83dc-803e, 'tcp://<IP_ADDRESS>:4567') Found matching local endpoint for a connection, blacklisting address tcp://<IP_ADDRESS>:4567
2023-08-02  7:09:04 0 [Note] WSREP: EVS version upgrade 0 -> 1
2023-08-02  7:09:04 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2023-08-02  7:09:04 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2023-08-02  7:09:04 0 [Note] WSREP: view(view_id(NON_PRIM,750c83dc-803e,1) memb {
        750c83dc-803e,0
} joined {
} left {
} partitioned {
})
2023-08-02  7:09:04 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50275S), skipping check
2023-08-02  7:09:34 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2023-08-02  7:09:34 0 [Note] WSREP: view((empty))
2023-08-02  7:09:34 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at ./gcomm/src/pc.cpp:connect():160
2023-08-02  7:09:34 0 [ERROR] WSREP: ./gcs/src/gcs_core.cpp:gcs_core_open():221: Failed to open backend connection: -110 (Connection timed out)
2023-08-02  7:09:35 0 [ERROR] WSREP: ./gcs/src/gcs.cpp:gcs_open():1669: Failed to open channel 'mariadb-operator' at 'gcomm://mariadb-galera-dev-0.mariadb-galera-dev-db-operator.svc.cluster.local,mariadb-galera-dev-1.mariadb-galera-dev-internal.mariadb-operator.svc.cluster.local,mariadb-galera-dev-2.mariadb-galera-dev-internal.marsvc.cluster.local': -110 (Connection timed out)
2023-08-02  7:09:35 0 [ERROR] WSREP: gcs connect failed: Connection timed out
2023-08-02  7:09:35 0 [ERROR] WSREP: wsrep::connect(gcomm://mariadb-galera-dev-0.mariadb-galera-dev-internal.mariadb-operator.svc.cluster.local,mariadb-galera-dev-1.mdev-internal.mariadb-operator.svc.cluster.local,mariadb-galera-dev-2.mariadb-galera-dev-internal.mariadb-operator.svc.cluster.local) failed: 7
2023-08-02  7:09:35 0 [ERROR] Aborting

Expected behaviour
Mariadb-operator could either

provide an automated recovery
provide a manual recovery procedure

Steps to reproduce the bug

Freshly deploy mariadb-operator (v0.19.0) using the following yaml:
apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera-dev
  namespace: mariadb-operator
spec:
  rootPasswordSecretKeyRef:
    name: mariadb
    key: root-password

  database: mariadb
  username: mariadb
  passwordSecretKeyRef:
    name: mariadb
    key: password

  connection:
    secretName: mariadb-conn
    secretTemplate:
      key: dsn

  image:
    repository: mariadb
    tag: "11.0.2"
    pullPolicy: IfNotPresent

  port: 3306

  replicas: 3

  podSecurityContext:
    runAsUser: 0

  galera:
    enabled: true
    sst: mariabackup
    replicaThreads: 1
    agent:
      image:
        repository: ghcr.io/mariadb-operator/agent
        tag: "v0.0.2"
        pullPolicy: IfNotPresent
      port: 5555
      kubernetesAuth:
        enabled: true
        authDelegatorRoleName: mariadb-galera-auth
      gracefulShutdownTimeout: 5s
    recovery:
      enabled: true
      clusterHealthyTimeout: 1m
      clusterBootstrapTimeout: 5m
      podRecoveryTimeout: 3m
      podSyncTimeout: 3m
    initContainer:
      image:
        repository: ghcr.io/mariadb-operator/init
        tag: "v0.0.5"
        pullPolicy: IfNotPresent
    volumeClaimTemplate:
      resources:
        requests:
          storage: 50Mi
      accessModes:
        - ReadWriteOnce

  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: "kubernetes.io/hostname"

  tolerations:
    - key: "mariadb.mmontes.io/ha"
      operator: "Exists"
      effect: "NoSchedule"

  podDisruptionBudget:
    maxUnavailable: 66%

  updateStrategy:
    type: RollingUpdate

  myCnf: |
    [mysqld]
    bind-address=<IP_ADDRESS>
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    max_allowed_packet=512M
    relay-log = relay-bin
    log-slave-updates = 1
    expire_logs_days = 7
    max_binlog_size = 1000M
    ## Safety
    ##
    skip_host_cache
    skip_name_resolve
    max_allowed_packet=16M
    max_connect_errors=1000000
    sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
    sysdate_is_now=1
    log_bin=mysql-bin
    expire_logs_days=14
    # Disabling for performance per http://severalnines.com/blog/9-tips-going-production-galera-cluster-mysql
    sync_binlog=0
    ## Caches and Limits
    tmp_table_size=32M
    max_heap_table_size=32M
    # Re-enabling as now works with Maria 10.1.2
    query_cache_type=1
    query_cache_limit=4M
    query_cache_size=256M
    max_connections=3000
    thread_handling = pool-of-threads
    thread_stack = 192K
    thread_cache_size = 4
    thread_pool_size = 8
    thread_cache_size=50
    thread_pool_oversubscribe = 3
    open_files_limit=65535
    table_definition_cache=4096
    table_open_cache=4096
    ## InnoDB
    #innodb=FORCE
    innodb_strict_mode=1
    innodb_autoinc_lock_mode=2
    innodb_doublewrite=1
    innodb_flush_method=O_DIRECT_NO_FSYNC
    innodb_log_files_in_group=2
    innodb_log_file_size=1G
    innodb_flush_log_at_trx_commit=2
    innodb_file_per_table=1
    # 80% Memory is default reco.
    # Need to re-evaluate when DB size grows
    innodb_buffer_pool_size=12G
    innodb_buffer_pool_instances=8
    innodb_file_format=Barracuda
    innodb_read_io_threads=8
    innodb_write_io_threads=4
    innodb_log_buffer_size = 512M

  resources:
#    requests:
#      cpu: 4000m
#      memory: 8Gi
    limits:
      memory: 32Gi

  volumeClaimTemplate:
    resources:
      requests:
        storage: 400Gi
    storageClassName: rook-ceph-block
    accessModes:
      - ReadWriteOnce

  service:
    type: ClusterIP

Additional context
We're using rook for storage provisioning. All disks report to be healthy and no crashes have been reported by ceph (or kubernetes) over the last 24 hours.
Environment details:

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:13:28Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}

mariadb-operator version: 0.19.0
Install method: helm
Install flavour: recommended

Hey there @tflabs-nl ! thanks for reporting this.
The operator should cover this situation by performing the cluster recovery, let's dig a bit deeper into this
For context, some users experienced similar problems, sometimes Galera requires extra configuration depending on the cluster, please take a look

https://github.com/mariadb-operator/mariadb-operator/issues/155
https://github.com/mariadb-operator/mariadb-operator/issues/159

2023-08-02  7:09:35 0 [ERROR] WSREP: ./gcs/src/gcs.cpp:gcs_open():1669: Failed to open channel 'mariadb-operator' at 'gcomm://mariadb-galera-dev-0.mariadb-galera-dev-db-operator.svc.cluster.local,mariadb-galera-dev-1.mariadb-galera-dev-internal.mariadb-operator.svc.cluster.local,mariadb-galera-dev-2.mariadb-galera-dev-internal.marsvc.cluster.local': -110 (Connection timed out)

To begin with, this connection timeout suggests that the peers of the cluster are unable to reach each other. Can you confirm that you have Endpoints by running the following command?
kubectl get endpoints mariadb-galera-internal -n default -o yaml

The next thing to do is checking why the operator is unable to bootstrap a new cluster, could you enable the debug logs and send the output? See this comment for reference

Thank you for your reply @mmontes11 !
~$ kubectl get endpoints -n mariadb-operator mariadb-galera-dev-internal -o yaml

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2023-08-02T14:36:53Z"
  labels:
    app.kubernetes.io/instance: mariadb-galera-dev
    app.kubernetes.io/name: mariadb
    service.kubernetes.io/headless: ""
  name: mariadb-galera-dev-internal
  namespace: mariadb-operator
  resourceVersion: "16906138"
  uid: c49cfbfa-18ba-4d5b-8ede-e15a5c86e5f1
subsets:
- addresses:
  - hostname: mariadb-galera-dev-1
    ip: <IP_ADDRESS>
    nodeName: c01-b01-am5-redacted-nl
    targetRef:
      kind: Pod
      name: mariadb-galera-dev-1
      namespace: mariadb-operator
      uid: d1d97e7b-7c4b-44bc-a6f0-82b0c591bb78
  - hostname: mariadb-galera-dev-0
    ip: <IP_ADDRESS>
    nodeName: c01-b03-am5-redacted-nl
    targetRef:
      kind: Pod
      name: mariadb-galera-dev-0
      namespace: mariadb-operator
      uid: 978f67fa-0446-479c-9ba2-58af8d22cfcc
  - hostname: mariadb-galera-dev-2
    ip: <IP_ADDRESS>
    nodeName: c01-b04-am5-redacted-nl
    targetRef:
      kind: Pod
      name: mariadb-galera-dev-2
      namespace: mariadb-operator
      uid: 9ef243b2-4eab-468a-8014-f1c7aaf6bbb8
  ports:
  - name: sst
    port: 4568
    protocol: TCP
  - name: ist
    port: 4567
    protocol: TCP
  - name: mariadb
    port: 3306
    protocol: TCP
  - name: agent
    port: 5555
    protocol: TCP
  - name: cluster
    port: 4444
    protocol: TCP
    ```

I decided to do a clean run to verify that it had nothing to do with an old version I upgraded from / me changing specs before the first init on all machines could complete. Unfortunately, the pod I was actively connected to during the import (mariadb-galera-dev-2) crashed during the new import.

Mysql Client reported: `ERROR 1180 (HY000) at line 1973: Got error 6 "No such device or address" during COMMIT`

Fortunately, this time, I had setup the ELK stack before any crashes on a brand new deployment. This log contains all the crashes ever since. Most interesting is what happened before `line 260` ( {"level":"info","ts":1690994798.0681577,"logger":"server","msg":"shutting   down server"} )

[Galera log view.csv](https://github.com/mariadb-operator/mariadb-operator/files/12244076/Galera.log.view.csv)

Hope this helps! Should I provide additional logs from the operator itself during the same time period?

By now all 3 of the replica's have crashed. Logs of the operator are full with:
{"level":"error","ts":1691061971.720084,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"22eb3850-91b4-4075-a039-efb80ee1ed69","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061971.7470686,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"cda8b669-530a-4a5f-b8e2-587d6640163c","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061971.7836084,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"506204ad-feee-4b41-8680-1625d873551b","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061971.835254,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"2019235e-01b0-4e28-abe2-52cf1c505824","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061971.9257667,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"9b13d159-8a5a-4df2-a4e7-83ca666767a0","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061972.1077244,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"50f0d2f9-dfc4-4f33-8ac6-5cd866804b47","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061972.4478083,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"5784aedd-ed2e-464f-b1fd-10336845ebc3","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061973.107079,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"7a9de322-2cc0-4cd2-aa80-221527e210f9","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061974.4047112,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"3f122380-a9ea-4e3f-88bf-4486292f8660","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061976.9765716,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"9d9f3de4-e88d-4693-89a2-1f391cde6478","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061982.1107109,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"7e31b09a-120d-48c9-ad47-0a5ba2707e37","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691061992.3668597,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"06c653c4-75f6-400e-b858-8f4c7d46b45e","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691062012.8634171,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"63aa7cf8-6135-4929-8012-7b2166d72942","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691062053.837543,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"2f8ce39e-54c3-4b1a-82b5-4a3a7c86b3f8","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691062135.772354,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"d3609911-749d-45dd-8ba3-36e3100c8fc0","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691062299.6243398,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"655eb929-5eb1-474a-92a3-7ba4e5889859","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691062627.3180552,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"d13d92bb-7936-4437-a3d2-9a43adb8f457","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
{"level":"error","ts":1691063282.6937213,"msg":"Reconciler error","controller":"connection","controllerGroup":"mariadb.mmontes.io","controllerKind":"Connection","connection":{"name":"mariadb-galera-dev","namespace":"mariadb-operator"},"namespace":"mariadb-operator","name":"mariadb-galera-dev","reconcileID":"31330894-0554-47fe-bb7c-e7312000ec1e","error":"MariaDB not ready","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}

Hey there @tflabs-nl ! Thanks a lot for the details
I was carefully taking a look to the logs you sent, but unfortunately they don't shed light about the question I want to answer, which is why the operator is not recovering the cluster? We know for sure that MariaDB is under pressure leading eventually to crashes and also getting out of sync because it cannot cope with the load. But this is a different topic, this requires tuning up spec.resources. I would suggest temporary increasing them to be able to restore the 13GB backup.

{"level":"info","ts":1690994798.0681577,"logger":"server","msg":"shutting down server"} )

These are the sidecar agent logs when the Pod gets killed, either by Kubernetes for reaching memory limits or because the spec.galera.recovery.podRecoveryTimeout is reached and therefore the operator attempts to restart the Pod.
Now, regarding why the operator is not recovering the cluster, we need more light to reach a conclusion, so I would suggest the following:

Get the Events of the MariaDB object:

kubectl get events --field-selector involvedObject.name=mariadb-galera -n mariadb --sort-by='.lastTimestamp'

Set operator log level to debug, like we suggested here, and get the operator logs when the Pods are in CrashLoopBackoff
Get the logs of all the MariaDB Pod container, not only of the main mariadb container but also the agent and init ones.

We experienced similar problems in the following issues, you may check them for context:

https://github.com/mariadb-operator/mariadb-operator/issues/155
https://github.com/mariadb-operator/mariadb-operator/issues/171
https://github.com/mariadb-operator/mariadb-operator/issues/159

As a takeaway from this issues, I will be adding a troubleshoot section to the GALERA.md

Hey there @tflabs-nl ! Thanks for investigating, we are making progress here, as this confirms that mariadb-operator is actually attempting to perform the recovery.

{"clusterBootstrapTimeout":"50m0s","clusterHealthyTimeout":"15m0s","enabled":true,"podRecoveryTimeout":"30m0s","podSyncTimeout":"30m0s"}

this sounds sensible, I actually increased the defaults yesterday as well

error unmarshaling bootstrap: unable to find uuid and seqno: uuid= seqno=

This is logged by the agent container when it attempts to parse the uuid and seqno from /var/lib/mysql/mariadb.err, here:
https://github.com/mariadb-operator/agent/blob/098241e9f786b87be539d70a5ecaa5b186dd64d7/pkg/galera/galera.go#L181
So basically the operator puts a Pod in recovery mode, meaning that it will be restarting and potentially outputing a line in
/var/lib/mysql/mariadb.err starting with WSREP: Recovered position:, which is what the agent is trying to parse. More info can be found at:
https://galeracluster.com/library/documentation/crash-recovery.html
The problem is that for some reason your MariaDB instance is unable to recover the uuid and seqno after setting the recovery mode, outputting something like this:
WSREP: Recovered position: 220dcdcb-1629-11e4-add3-aec059ad3734:1122

Instead, your mariadb.err must be empty or corrupt, leading to the following error:
error unmarshaling bootstrap: unable to find uuid and seqno: uuid=<nil> seqno=<nil>

It would be great if you could grab /var/lib/mysql/mariadb.err to see what we are dealing with.
Maybe it has to do with the configuration somehow?
    [mysqld]
    bind-address=<IP_ADDRESS>
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    max_allowed_packet=512M
    relay-log = relay-bin
    log-slave-updates = 1
    expire_logs_days = 7
    max_binlog_size = 1000M
    ## Safety
    ##
    skip_host_cache
    skip_name_resolve
    max_allowed_packet=16M
    max_connect_errors=1000000
  sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
    sysdate_is_now=1
    log_bin=mysql-bin
    expire_logs_days=14
    # Disabling for performance per http://severalnines.com/blog/9-tips-going-production-galera-cluster-mysql
    sync_binlog=0
    ## Caches and Limits
    tmp_table_size=32M
    max_heap_table_size=32M
    # Re-enabling as now works with Maria 10.1.2
    query_cache_type=1
    query_cache_limit=4M
    query_cache_size=256M
    max_connections=3000
    thread_handling = pool-of-threads
    thread_stack = 192K
    thread_cache_size = 4
    thread_pool_size = 8
    thread_cache_size=50
    thread_pool_oversubscribe = 3
    open_files_limit=65535
    table_definition_cache=4096
    table_open_cache=4096
    ## InnoDB
    #innodb=FORCE
    innodb_strict_mode=1
    innodb_autoinc_lock_mode=2
    innodb_doublewrite=1
    innodb_flush_method=O_DIRECT_NO_FSYNC
    innodb_log_files_in_group=2
    innodb_log_file_size=1G
    innodb_flush_log_at_trx_commit=2
    innodb_file_per_table=1
    # 80% Memory is default reco.
    # Need to re-evaluate when DB size grows
    innodb_buffer_pool_size=12G
    innodb_buffer_pool_instances=8
    innodb_file_format=Barracuda
    innodb_read_io_threads=8
    innodb_write_io_threads=4
    innodb_log_buffer_size = 512M

cc @grooverdan in case you have any ideas. sync_binlog=0 might not be helping with the consistency.

What is the best way to bootstrap a cluster in which all nodes failed using the operator?

to answer your question, recovering a cluster with all failed nodes is what the operator is attempting to do here, but it needs the seqnos and uuids to know which node should be used to bootstrap a new cluster.

Hi @mmontes11 ! I have to agree with you that disabling binlog syncing is not the best in terms of consistency.
I saw a blog post regarding mariadb galera optimizations and followed that.
Said blog post
I first had some questions regarding the problems that may arise from not enabling sync_binlog but found the following blog post:
trx commit 2 /w sync binlog evil
Thereby figuring that 'a couple transactions' would be fine, in our case. (As I understood, this means a couple of transactions on pod failure, which should be rare)
I seem to be unable to overwrite the arguments of the agent. (spec: Forbidden: pod updates may not change fields other than... Also the agent doesn't have bash or sh, so I cannot execute into the pod. The mariadb container does not seem to be started, so I cannot execute any commands in there either. Maybe it would be an idea to create a second image that is based on ubuntu or any other image that has some basic support for debugging?
Is there any other way I can cat the /var/lib/mysql/mariadb.err file without having to destroy (or in this case possibly rescale) the mariadb config and mounting the PVC in another pod?

Hey @tflabs-nl !

Maybe it would be an idea to create a second image that is based on ubuntu or any other image that has some basic support for debugging?

This will work for sure

Is there any other way I can cat the /var/lib/mysql/mariadb.err file without having to destroy (or in this case possibly rescale) the mariadb CR and mounting the PVC in another pod?

I was giving ephemeral containers  a try, but unfortunatelly AFAIK they don't support volumeMounts:
 kubectl debug -it mariadb-galera-0  --image=busybox --copy-to=mariadb-debug

As an alternative, you can open a debugging shell on the Node with the filesystem mounted at /host :
kubectl debug node/mdb-worker3 -it --image=busybox

Hi @mmontes11 !
I tried to access the container's filesystem via a the host by looking up the Container ID via kubectl describe pod <podname> and then searching for that container ID within the /proc/mounts file on the host.
This yielded a brief result for the MariaDB container, but because it fails, the FS is quickly unmounted. Accessing the agent container for example is no problem, but AFAIK there's nothing of interest in there, right?
I'll try to do a race with kubernetes by trying to loop through all the containers with cat <containerID>/var/lib/mysql/mariadb.err from the host. Hopefully one of them is a hit. Would be cool to see if that would work.

Hey @tflabs-nl !

Accessing the agent container for example is no problem, but AFAIK there's nothing of interest in there, right?

The agent shares the /var/lib/mysql volume with the mariadb one, so it might be valid as well.

Hi @mmontes11 , seems like /var/lib/mysql does not exist on that mountpoint. Which also kind of makes sense, as the PVC storage-mariadb-galera-dev-2 as well as galera-mariadb-galera-dev-2 are both RWO. The path I was looking at also only contains the rootfs of the container, so no mounts included.
Eventually, I did find the correct mount point location. Getting the logs was as easy as SSH'ing into the node that currently runs (or is supposed to run) the pod and issue the following command:
cat $(find . -name "pvc-562ce812-e4bb-49a3-8d00-6a8963c017cf")/mount/mariadb.err
in which pvc-562ce812-e4bb-49a3-8d00-6a8963c017cf is the full PVC name as obtained by kubectl get pvc -n mariadb-operator
Here is a small portion of the log:
2023-08-05  6:04:40 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 88
2023-08-05  6:04:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-08-05  6:04:40 0 [Note] InnoDB: Number of transaction pools: 1
2023-08-05  6:04:40 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-08-05  6:04:40 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-08-05  6:04:40 0 [Note] InnoDB: Initializing buffer pool, total size = 12.000GiB, chunk size = 192.000MiB
2023-08-05  6:04:40 0 [Note] InnoDB: Completed initialization of buffer pool
2023-08-05  6:04:40 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2023-08-05  6:04:41 0 [Note] InnoDB: Opened 3 undo tablespaces
2023-08-05  6:04:41 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2023-08-05  6:04:41 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-08-05  6:04:41 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-08-05  6:04:41 0 [Note] InnoDB: log sequence number 37745; transaction id 4
2023-08-05  6:04:41 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2023-08-05  6:04:41 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-05  6:04:41 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-08-05  6:04:41 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2023-08-05  6:04:41 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:04:41 0 [Warning] 'innodb-buffer-pool-instances' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:04:41 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:04:41 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
2023-08-05  6:04:41 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2023-08-05  6:04:41 0 [Note] WSREP: disabled, skipping position recovery
Warning: Memory not freed: 280
2023-08-05  6:06:47 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 88
2023-08-05  6:06:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-08-05  6:06:47 0 [Note] InnoDB: Number of transaction pools: 1
2023-08-05  6:06:47 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-08-05  6:06:47 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-08-05  6:06:47 0 [Note] InnoDB: Initializing buffer pool, total size = 12.000GiB, chunk size = 192.000MiB
2023-08-05  6:06:47 0 [Note] InnoDB: Completed initialization of buffer pool
2023-08-05  6:06:47 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2023-08-05  6:06:48 0 [Note] InnoDB: Opened 3 undo tablespaces
2023-08-05  6:06:48 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2023-08-05  6:06:48 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-08-05  6:06:48 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-08-05  6:06:48 0 [Note] InnoDB: log sequence number 37745; transaction id 4
2023-08-05  6:06:48 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2023-08-05  6:06:48 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-05  6:06:48 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-08-05  6:06:48 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2023-08-05  6:06:48 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:06:48 0 [Warning] 'innodb-buffer-pool-instances' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:06:48 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:06:48 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
2023-08-05  6:06:48 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2023-08-05  6:06:48 0 [Note] WSREP: disabled, skipping position recovery
Warning: Memory not freed: 280
2023-08-05  6:10:15 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 88
2023-08-05  6:10:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-08-05  6:10:15 0 [Note] InnoDB: Number of transaction pools: 1
2023-08-05  6:10:15 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-08-05  6:10:15 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-08-05  6:10:15 0 [Note] InnoDB: Initializing buffer pool, total size = 12.000GiB, chunk size = 192.000MiB
2023-08-05  6:10:15 0 [Note] InnoDB: Completed initialization of buffer pool
2023-08-05  6:10:15 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2023-08-05  6:10:16 0 [Note] InnoDB: Opened 3 undo tablespaces
2023-08-05  6:10:16 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2023-08-05  6:10:16 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-08-05  6:10:16 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-08-05  6:10:16 0 [Note] InnoDB: log sequence number 37745; transaction id 4
2023-08-05  6:10:16 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2023-08-05  6:10:16 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-05  6:10:16 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-08-05  6:10:16 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2023-08-05  6:10:16 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:10:16 0 [Warning] 'innodb-buffer-pool-instances' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:10:16 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:10:16 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
2023-08-05  6:10:16 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2023-08-05  6:10:16 0 [Note] WSREP: disabled, skipping position recovery
Warning: Memory not freed: 280
2023-08-05  6:15:50 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 88
2023-08-05  6:15:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-08-05  6:15:50 0 [Note] InnoDB: Number of transaction pools: 1
2023-08-05  6:15:50 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-08-05  6:15:50 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-08-05  6:15:50 0 [Note] InnoDB: Initializing buffer pool, total size = 12.000GiB, chunk size = 192.000MiB
2023-08-05  6:15:50 0 [Note] InnoDB: Completed initialization of buffer pool
2023-08-05  6:15:50 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2023-08-05  6:15:51 0 [Note] InnoDB: Opened 3 undo tablespaces
2023-08-05  6:15:51 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2023-08-05  6:15:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-08-05  6:15:51 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-08-05  6:15:51 0 [Note] InnoDB: log sequence number 37745; transaction id 4
2023-08-05  6:15:51 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2023-08-05  6:15:51 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-05  6:15:51 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-08-05  6:15:51 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2023-08-05  6:15:51 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:15:51 0 [Warning] 'innodb-buffer-pool-instances' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:15:51 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:15:51 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
2023-08-05  6:15:51 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2023-08-05  6:15:51 0 [Note] WSREP: disabled, skipping position recovery
Warning: Memory not freed: 280
2023-08-05  6:21:32 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 88
2023-08-05  6:21:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-08-05  6:21:32 0 [Note] InnoDB: Number of transaction pools: 1
2023-08-05  6:21:32 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-08-05  6:21:32 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-08-05  6:21:32 0 [Note] InnoDB: Initializing buffer pool, total size = 12.000GiB, chunk size = 192.000MiB
2023-08-05  6:21:32 0 [Note] InnoDB: Completed initialization of buffer pool
2023-08-05  6:21:32 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2023-08-05  6:21:33 0 [Note] InnoDB: Opened 3 undo tablespaces
2023-08-05  6:21:33 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2023-08-05  6:21:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-08-05  6:21:33 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-08-05  6:21:33 0 [Note] InnoDB: log sequence number 37745; transaction id 4
2023-08-05  6:21:33 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2023-08-05  6:21:33 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-05  6:21:33 0 [Note] Plugin 'wsrep-provider' is disabled.
2023-08-05  6:21:33 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
2023-08-05  6:21:33 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:21:33 0 [Warning] 'innodb-buffer-pool-instances' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:21:33 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
2023-08-05  6:21:33 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
2023-08-05  6:21:33 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2023-08-05  6:21:33 0 [Note] WSREP: disabled, skipping position recovery
Warning: Memory not freed: 280

I don't see any UUID's being mentioned unfortunately...

Hey @tflabs-nl ! thanks a lot for your effort diagnosing this. This is very valuable not only for your use case but also for the troubleshooting section I'm currently writing.

I don't see any UUID's being mentioned unfortunately...

right, but then this explains why the agent fails trying to recover them, returning the error:
error unmarshaling bootstrap: unable to find uuid and seqno: uuid=<nil> seqno=<nil>

these log lines might be pointing the root cause:
2023-08-05  6:15:51 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2023-08-05  6:15:51 0 [Note] WSREP: disabled, skipping position recovery

after quickly googling them, I've seen other people reporting similar problems:

https://github.com/ClusterLabs/resource-agents/issues/940
https://dba.stackexchange.com/questions/134762/systemctl-startup-script-gives-wsrep-failed-to-recover-position-manual-star

will dig deeper and get back to you
cc @grooverdan
Thanks a lot! 🙏🏻

This is how a healthy /var/lib/mysql/mariadb.err looks like:
root@mariadb-galera-0:/# cat /var/lib/mysql/mariadb.err
2023-08-05 14:59:14 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 1
2023-08-05 14:59:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-08-05 14:59:14 0 [Note] InnoDB: Number of transaction pools: 1
2023-08-05 14:59:14 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-08-05 14:59:14 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-08-05 14:59:14 0 [Note] InnoDB: Using liburing
2023-08-05 14:59:14 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2023-08-05 14:59:14 0 [Note] InnoDB: Completed initialization of buffer pool
2023-08-05 14:59:14 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2023-08-05 14:59:14 0 [Note] InnoDB: Opened 3 undo tablespaces
2023-08-05 14:59:14 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2023-08-05 14:59:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2023-08-05 14:59:14 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2023-08-05 14:59:14 0 [Note] InnoDB: log sequence number 81542; transaction id 95
2023-08-05 14:59:14 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2023-08-05 14:59:14 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-05 14:59:15 0 [Note] Server socket created on IP: '<IP_ADDRESS>'.
2023-08-05 14:59:15 0 [Note] WSREP: Recovered position: 6ea235ec-3232-11ee-8152-4af03d2c43a9:17
Warning: Memory not freed: 280

I've spotted another difference:
2023-08-05  6:21:33 0 [Note] Plugin 'wsrep-provider' is disabled.

Which is trange, because we do enable wsrep explicitly in /etc/mysql/mariadb.conf.d/0-gakera.cnf:
https://github.com/mariadb-operator/init/blob/main/pkg/config/config.go#L46
Maybe this is clashing with your spec.myCnf?

Maybe this is clashing with your spec.myCnf?

I've checked this, and most likely it shouldn't be the case, see:

https://mariadb.com/kb/en/configuring-mariadb-galera-cluster/

[Note] Plugin 'wsrep-provider' is disabled.
[ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded

My understanding is that somehow the mysql.plugin table got corrupted somehow and the wsrep-provider plugin wasn't loaded. Unfortunately Galera relies on this and without it recovering the sequence is not possible.
Maybe it is worth starting from a clean Database at this point if it's not production? Seen some hacks to solve this though:

https://stackoverflow.com/questions/47075429/error-setting-up-mysql-table-mysql-plugin-doesnt-exist

In production, the thing will be taking Backups regularly enough to minimize RTO and RPO, and Restore them if this situation happens to bring mysql.plugin back to its operational state.

Hey @mmontes11 ! Had holiday and tried to do a bit of pc detoxing haha. Would it be beneficial for the troubleshooting documentation if I tried that 'hack'? If not I will recreate the environment from scratch and re-import without disabling the sync_binlog.
Maybe a bit of hijacking my own issue, but does backup / restore do incremental backups?
Thank you so much for creating these troubleshooting steps!

I'm not quite sure about this hack, probably doesn't worth it, but I leave it up to you. Even if it helped to fix the mysql.plugin table it wouldn't be something to be implemented in the operator, as automating it would bring more problems than benefits problably.

If not I will recreate the environment from scratch and re-import without disabling the sync_binlog.

This would be my take if you ask me

Maybe a bit of hijacking my own issue, but does backup / restore do incremental backups?

We don't support incremental backups but it is in our roadmap:

https://github.com/mariadb-operator/mariadb-operator/issues/17

Before we do that, we are planning to support S3 storage for backups:

https://github.com/mariadb-operator/mariadb-operator/issues/6

Thank you so much for creating these troubleshooting steps!

Thanks, it hopefully shed some light in your investigation.


How to describe request json body in depth?

Hm?!
https://github.com/swagger-api/swagger-core/issues/1723

@ivansky - that's not really a way to open tickets that encourage discussions. If you'd like to raise an issue and discuss possible changes, please do so in a manner that helps the community, don't just send people chasing threads on other projects.

Closing as tackled in other tickets.


Issuing credits to Evan

Issue 10000 credits to @gnomevan.

 Follow up
On Monday, 18-Mar-24 07:24:51 UTC by Gov4Git dev
Issued 10000 credits to member @gnomevan.


It is downloading the .dev version instead of 2.7.9[Question]

It is downloading the .dev version instead of 2.7.9 when informing in flutter, that is, I inform 2.7.9 and download 2.7.10-dev.1

Interesting, so in your pubspec.yaml file you have pro_image_editor: 2.7.9, but it installs version 2.7.10-dev.1, right?
Can you try to run flutter clean first and check if it installs the correct version.
And how do you know that the dev version is installed and not 2.7.9?

Right yes, I've already run flutter clean and then flutter pub get, as I know you're downloading the version I mentioned, follow below

Okay, I got it. I also tried to reproduce it, but in my case it always takes the correct version. It's also special that it downloads the version 2.7.10-dev.1 because this is an older version from the dev branch. I already released in the dev branch version 3.0.0-dev.3. Anyway, are you using flutter 3.22.0 or an older version of it? And do you use the stable channel from flutter?

I'm currently using version 3.19.6 of flutter, the question is I have a lib of mine which is a ui package so that my app uses everything from this lib, so I'm setting the pro editor in my ui package and pp consuming from there, and that's why the problem seems to be occurring. I'll try to use it directly in the app even without going through the UI package.

Sounds interesting. Is your UI package public or private? If it's public, I can test it if I have the same problem.
Just to confirm, you're also running flutter clean inside your UI package and not using the 2.7.10-dev.1 version there, right?

Exactly, I'm using 2.7.8 in my UI package, I apply flutter clean and flutter pub get, the package is private, as it is a company application, we work with monitoring works and we need a photo editor, I found the its very good indeed.

So you now install version 2.7.8 in your project, and inside your UI package you still have the editor installed? Because it seems like there are different versions of the editor in conflict, or maybe there is a conflict with the image package. Are you also installing the image package inside your current package or UI package? If so, which version are you using? And when you run flutter pub get, do you see any warnings/errors in the terminal.

Hi, perfect, in fact it was the image version, yesterday at the end of the day I ended up detecting this, I really appreciate your help and your commitment to helping me. The package is very good, congratulations on the excellent work.
I'm just having difficulty using it within my UI package, I'm going to use it in the application right now, what happens is, when I close the editor, the loading of saving changes remains on the screen, if I put a Navigator.of(context ).pop() in the onCloseEditor method it ends up closing the current screen, and the loading remains, I tried putting Navigator.of(context).pop() in onImageEditingComplete and the same thing happens, remembering that this behavior occurs when using the pro editor In my UI package, the app works normally.

  Segue o código chamador do pro editor(componente)

Fiz um app teste e nele funciona perfeitamente.

In most cases, the problem is that people are using a special navigation system like the one in the get package.
The other case is that something is wrong with the configurations. Can you post me the ProImageEditor widget, so I can take a look at it, maybe I can see the issue then. For example, ProImageEditor.network(/*config stuff*/).

Ola amigo, realmente implementei o pro image editor em um outro projeto para teste utilizando a minha biblioteca de ui contendo o seu projeto e tudo funcionou normalmente.  No app da empresa em principio usamos no Navigator, mas vou verificar todo o fluxo para ver onde está o problema.  Agradeço a sua ajuda.
Uma dica apenas, quero mudar a cor do systemNavigationBarColor mas em iOS e não consigo, não surte o efeito desejado. segue abaixo o código utilizado para fazer isto:
  @override   Widget build(BuildContext context) {     SystemChrome.setSystemUIOverlayStyle(SystemUiOverlayStyle.dark.copyWith(       systemNavigationBarColor: CinUiFoundationColor.appBarColorBlack,       statusBarColor: CinUiFoundationColor.appBarColorBlack,       systemNavigationBarDividerColor: CinUiFoundationColor.appBarColorBlack,       statusBarBrightness: Brightness.dark,     ));     return !widget.params.isImageEditor         ? CinUiScaffoldOrganism(             appBar: CinUiAppBarMolecule(               backgroundColor: CinUiFoundationColor.appBarColorBlack,               leadingIcon: CinUiFoundationIcon.close,               useLeadingIconBorder: false,               leadingIconColor: CinUiFoundationColor.appBarColor,               leadingIconSize: CinUiFoundationSize.iconDefaultSize,               onLeadingPressed: () => Navigator.of(context).pop(),             ),             body: CinUiBodyMolecule(               params: CinUiBodyParams(                 backgroundColor: CinUiFoundationColor.bodyContentBackgroundLiteGray,                 useTopBorderDefault: false               ),               child: CinOrganismImageEditor(params: widget.params),             ),           )         : CinOrganismImageEditor(             params: widget.params,             onComplete: (bytes) async {               imageBytes = bytes;             },             onCloseEditor: () {               Navigator.of(context).pop(imageBytes);             },           );   }

Ola amigo,
Inside the code I use AnnotatedRegion<SystemUiOverlayStyle> to set these settings. You can try to edit your configs from the image editor like below that should work:
ProImageEditor.asset(
    'demo.png',
    configs: ProImageEditorConfigs(
      imageEditorTheme: ImageEditorTheme(
        uiOverlayStyle: SystemUiOverlayStyle.dark.copyWith(
            // your configs...
            ),
      ),
  ),

Perfect, good my friend, as I reported above, using the pro editor through the UI library in a test project, it worked, but in the app project the reported problem still persists, I know it is already helping me a lot, but there would be a way to have more direct contact, to talk and be able to show the code and maybe then you can help me maybe detect the problem, which in fact I can't understand what happens and only in the app itself.
I checked the route issue and we only use Flutter's navigator, it doesn't have any pakage like get for navigation.
Thank you in advance for all your help.

I identified the problem, we used the package
https://pub.dev/packages/flutter_screenutil
to be able to adapt the screen to tablet issues and others, and otherwise it works, but then we have the problem of responsiveness for tablets.

Okay, this is interesting. I will try it in my example when I have time and if I can reproduce it, I will also try to fix it. Do I need to configure the ScreenUtilInit in a special way, or is it enough to just wrap all content in it?

I just tried to reproduce it, but in my case it still works even with this package. Can you post here for me a simple example code how I can reproduce it.

Hello, friend, first of all I apologize, because in reality it is not this package that causes the problems, but this one, asuka
https://pub.dev/packages/asuka
I will send the code we have with both packages in the main configuration

When removing the line "child: Asuka.builder(context, widget)", from the image below, just returning the builder "widget", it works normally.
I'm on flutter version 3.19.6 with editor version 2.7.8

Okay, I think I found the problem. The package Asuka needs the build context from the dialog itself to close the same dialog. I updated the loading-dialog that it first tries to use the own context before it takes the global context to close it.
I released version 2.7.10, can you try it if it works with this version?

Oops I forgot that you said you use 3.19.6🙈 Actually I only support the newest flutter version, but I released for you version 2.7.8-fix-1 now which also supports 3.19.6.

Thank you very much, from what I saw it worked well, I will do more tests.

Can you check inside the ProImageEditorConfigs and then imageEditorTheme  if there is set any colors for the appbar? The code in the editor looks like that:
AppBar(
      automaticallyImplyLeading: false,
      foregroundColor: imageEditorTheme.appBarForegroundColor,
      backgroundColor: imageEditorTheme.appBarBackgroundColor,
      /// other stuff
)

Very good friend, perfect, resolved, I really appreciate all the attention
and help. Very good to the editor
Em ter., 21 de mai. de 2024 às 13:38, Alex Frei @.***>
escreveu:

Can you check inside the ProImageEditorConfigs and then imageEditorTheme
if there is set any colors for the appbar? The code in the editor looks
like that:
AppBar(
automaticallyImplyLeading: false,
foregroundColor: imageEditorTheme.appBarForegroundColor,
backgroundColor: imageEditorTheme.appBarBackgroundColor,
/// other stuff)
—
Reply to this email directly, view it on GitHub
https://github.com/hm21/pro_image_editor/issues/48#issuecomment-2123022342,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOQ2JI4EU57PEG7ZHD7IJRDZDNZ7RAVCNFSM6AAAAABH3IEWY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGAZDEMZUGI
.
You are receiving this because you authored the thread.Message ID:
@.***>


Authentication and/or multi-tenancy

Is your feature request related to a problem? Please describe.
not problem related
Describe the solution you'd like
adding a signin process to the code to allow multiusers or multitenancy (many customers)
Describe alternatives you've considered
reverse proxy authentication or custom app in front of spoolman
Additional context
n/a

+1. I'd like to be able to access this from outside my network, but I can't expose it without some sort of authentication


Add Discord Link to README

Changelog

Add invite link to #blueprinter Discord channel in README.

Checklist:

[ ] I have updated the necessary documentation
[ ] I have signed off all my commits as required by DCO
[ ] My build is green

Alternative if we want to keep it simple and not use the svg that's a bit big.  Looks good on Org homepage but maybe not here.  Or do want to attract people's attention.  wdyt?
https://img.shields.io/badge/Chat-EDEDED?logo=discord&style=for-the-badge
results in

Alternative if we want to keep it simple and not use the svg that's a bit big. Looks good on Org homepage but maybe not here. Or do want to attract people's attention. wdyt?

@jmeridth Yeah that's a good call. The bigger icon looks a bit off here!


Add unit test to check if quotes have desired formats.

Add mocha unit tests to check if quotes have desired formats.
{
      "quote": "String",
      "movie": "String",
      "type": "String",
      "year": Number
}

The test should check if the quote exists and is a type of String, same goes with each attribute with an associated data type.

I would like to do that. Can you assign this issue for me?

@Laviitz fixed this issue, just need some small changes.


[Bug fix] SageMaker case sensitivity

TL;DR
A lot of the fields in SageMaker CRDs are case sensitive, but only some of them are validated at CRD level. The bug this PR fixed was detected by the operator.
Type

[x] Bug Fix
[ ] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[x] Smoke tested
[ ] Unit tests added
[ ] Code documentation added
[ ] Any pending items have an associated Issue

Complete description
A lot of the fields in SageMaker CRDs are case sensitive, but only some of them are validated at CRD level. The bug this PR fixed was detected by the operator. We use strings.Title(strings.ToLower(<protoc enum>.String())) to fix it.
Tracking Issue
Follow-up issue
NA

Codecov Report

Merging #107 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #107   +/-   ##
=======================================
  Coverage   55.53%   55.54%
=======================================
  Files          99       99
  Lines        5913     5914    +1
=======================================
+ Hits         3284     3285    +1
  Misses       2318     2318
  Partials      311      311

Impacted Files
Coverage Δ

go/tasks/plugins/k8s/sagemaker/utils.go
50.33% <100.00%> (+0.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44216c3...242e183. Read the comment docs.

Have we tested sufficient surface area, that this is not an issue with other enums? All Protobuf enums are capitalized, i would presume all of them would have a problem? no?

👍

@kumare3 I’ve double checked that, except for the “InputContentType” enum which requires a explicit translation, all the other enums that I use in SageMaker plugin are wrapped inside the same strings.Title(strings.ToLower()) thing


Request for Mixtral 8X7B inference with DP+EP+TP

I want to use the mixtral 8X7B model for inference, but currently it only supports autoTP. How to add more support to enable it to use more parallelism (e.g. EP, DP)

I am experiencing the same issue with enabling TP+EP for inference on the Mixtral model（or other MoE model）. Have you managed to find a solution to this problem?

I only inference with TP+EP using GPT-MoE64, not for Mixtral. And  the script has to utilize the Megatron-Deepspeed.

I am experiencing the same issue with enabling TP+EP for inference on the Mixtral model（or other MoE model）. Have you managed to find a solution to this problem?

I only inference with TP+EP using GPT-MoE64, not for Mixtral. And the script has to utilize the Megatron-Deepspeed.

I am experiencing the same issue with enabling TP+EP for inference on the Mixtral model（or other MoE model）. Have you managed to find a solution to this problem?

I only inference with TP+EP using GPT-MoE64, not for Mixtral. And the script has to utilize the Megatron-Deepspeed.

I'm facing some difficulties with getting EP to work correctly. Could you share how you managed to implement TP + EP on GPT-MoE64? Any relevant scripts or code, suggestions would be greatly appreciated. If it's convenient, we can also discuss this further via email. My email address is<EMAIL_ADDRESS>

I am experiencing the same issue with enabling TP+EP for inference on the Mixtral model（or other MoE model）. Have you managed to find a solution to this problem?

I only inference with TP+EP using GPT-MoE64, not for Mixtral. And the script has to utilize the Megatron-Deepspeed.

I'm facing some difficulties with getting EP to work correctly. Could you share how you managed to implement TP + EP on GPT-MoE64? Any relevant scripts or code, suggestions would be greatly appreciated. If it's convenient, we can also discuss this further via email. My email address is<EMAIL_ADDRESS>
All the scripts can be found in Megatron-DeepSpeed/examples_deepspeed. The scripts for training are in MoE/*.sh, e.g., ds_pretrain_gpt_1.3B_MoE128.sh. The inference script is generate_text.sh. But for inference, there are many bugs in the relevant  codes. Usually, you can fix these bugs by delete the unnecessary params.

I am experiencing the same issue with enabling TP+EP for inference on the Mixtral model（or other MoE model）. Have you managed to find a solution to this problem?

I only inference with TP+EP using GPT-MoE64, not for Mixtral. And the script has to utilize the Megatron-Deepspeed.

I'm facing some difficulties with getting EP to work correctly. Could you share how you managed to implement TP + EP on GPT-MoE64? Any relevant scripts or code, suggestions would be greatly appreciated. If it's convenient, we can also discuss this further via email. My email address is<EMAIL_ADDRESS>
All the scripts can be found in Megatron-DeepSpeed/examples_deepspeed. The scripts for training are in MoE/*.sh, e.g., ds_pretrain_gpt_1.3B_MoE128.sh. The inference script is generate_text.sh. But for inference, there are many bugs in the relevant codes. Usually, you can fix these bugs by delete the unnecessary params.

Thank you so much for your response. I followed your approach for training and inference, but I encountered an issue where the trained weights seem to be inconsistent with what is required by the generate_text.sh script.
I’m currently facing an issue with the generate_text.sh script not being able to find the XX/iter_0000200/mp_rank_00/model_optim_rng.pt file. The weights generated from the ds_pretrain_gpt_125M_MoE64.sh script are saved as XX/global_step200/*, as shown in the image below.

I suspect that the weights might need to be converted from Megatron-DeepSpeed format to Megatron format. I attempted to use the tools/convert_checkpoint/deepspeed_to_megatron.py  for this conversion, but it seems that this file is not compatible with MoE models.
Have you encountered a similar issue? If so, could you please share how you managed to resolve it?
Thank you very much for your assistance.

I am experiencing the same issue with enabling TP+EP for inference on the Mixtral model（or other MoE model）. Have you managed to find a solution to this problem?

I only inference with TP+EP using GPT-MoE64, not for Mixtral. And the script has to utilize the Megatron-Deepspeed.

I'm facing some difficulties with getting EP to work correctly. Could you share how you managed to implement TP + EP on GPT-MoE64? Any relevant scripts or code, suggestions would be greatly appreciated. If it's convenient, we can also discuss this further via email. My email address is<EMAIL_ADDRESS>
All the scripts can be found in Megatron-DeepSpeed/examples_deepspeed. The scripts for training are in MoE/*.sh, e.g., ds_pretrain_gpt_1.3B_MoE128.sh. The inference script is generate_text.sh. But for inference, there are many bugs in the relevant codes. Usually, you can fix these bugs by delete the unnecessary params.

Thank you so much for your response. I followed your approach for training and inference, but I encountered an issue where the trained weights seem to be inconsistent with what is required by the generate_text.sh script. I’m currently facing an issue with the generate_text.sh script not being able to find the XX/iter_0000200/mp_rank_00/model_optim_rng.pt file. The weights generated from the ds_pretrain_gpt_125M_MoE64.sh script are saved as XX/global_step200/*, as shown in the image below.  I suspect that the weights might need to be converted from Megatron-DeepSpeed format to Megatron format. I attempted to use the tools/convert_checkpoint/deepspeed_to_megatron.py for this conversion, but it seems that this file is not compatible with MoE models. Have you encountered a similar issue? If so, could you please share how you managed to resolve it? Thank you very much for your assistance.

Did you have solved the issue? For inference, the model_optim_rng.pt file is not required. Maybe you can try the function load_model_weights_only in the "megatron/training.py". Moreover, it's better to load the model weights and resharding them into the target by yourself, the original scripts can inference using the same parallelism with training.

I am experiencing the same issue with enabling TP+EP for inference on the Mixtral model（or other MoE model）. Have you managed to find a solution to this problem?

I only inference with TP+EP using GPT-MoE64, not for Mixtral. And the script has to utilize the Megatron-Deepspeed.

I'm facing some difficulties with getting EP to work correctly. Could you share how you managed to implement TP + EP on GPT-MoE64? Any relevant scripts or code, suggestions would be greatly appreciated. If it's convenient, we can also discuss this further via email. My email address is<EMAIL_ADDRESS>
All the scripts can be found in Megatron-DeepSpeed/examples_deepspeed. The scripts for training are in MoE/*.sh, e.g., ds_pretrain_gpt_1.3B_MoE128.sh. The inference script is generate_text.sh. But for inference, there are many bugs in the relevant codes. Usually, you can fix these bugs by delete the unnecessary params.

Thank you so much for your response. I followed your approach for training and inference, but I encountered an issue where the trained weights seem to be inconsistent with what is required by the generate_text.sh script. I’m currently facing an issue with the generate_text.sh script not being able to find the XX/iter_0000200/mp_rank_00/model_optim_rng.pt file. The weights generated from the ds_pretrain_gpt_125M_MoE64.sh script are saved as XX/global_step200/*, as shown in the image below.  I suspect that the weights might need to be converted from Megatron-DeepSpeed format to Megatron format. I attempted to use the tools/convert_checkpoint/deepspeed_to_megatron.py for this conversion, but it seems that this file is not compatible with MoE models. Have you encountered a similar issue? If so, could you please share how you managed to resolve it? Thank you very much for your assistance.

Did you have solved the issue? For inference, the model_optim_rng.pt file is not required. Maybe you can try the function load_model_weights_only in the "megatron/training.py". Moreover, it's better to load the model weights and resharding them into the target by yourself, the original scripts can inference using the same parallelism with training.

Hi, I’m sorry for the late reply. I haven’t solved the issue, but I was able to achieve the desired outcome using another method. I will definitely try out the approach you mentioned as well. Thank you so much for your help!

That's great.


[Feature] add time function: makedate #13051

What type of PR is this：

[ ] BugFix
[X] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

Checklist:

[X] I have added test cases for my bug fix or my new feature
[X] This pr will affect users' behaviors
[X] This pr needs user documentation (for new or modified features or behaviors)
[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[ ] I have checked the version labels which the pr will be auto backported to target branch

[ ] 3.0
[ ] 2.5
[ ] 2.4
[ ] 2.3

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.chunyang.zhang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

this one is abandoned，remove if possible


bug: Using a local path e.g. copier that-template a-new-folder still uses git if that-template is a repo

Describe the problem
Perhaps this isn't a bug and the workflow needs documentation. Underlying issue is "how do you test templates locally without committing the changes"?
To Reproduce

Building or extending a template and want to test locally
Inside the that-template folder
Run copier . ./tmp

Expected behavior
I would expect copier to use the local changes instead of the git repo. Perhaps better illustrated in the screen below where I modified autoprettier to have an invalid config (using string not str - a real world example 🤦♂).
I would expect copier to use the HEAD only when I specify the full git url e.g. copier copy https://github.com/copier-org/autopretty.git . (or one of the gh url shortcuts)
Screenshots/screencasts

Environment

OS: Windows
Copier version: copier 5.1.0
Python version: 3.8.5
Installation method: pipx+pypi

Additional context
Perhaps this isn't a bug and the workflow needs documentation

Hello, thanks for contributing this issue. Closing as duplicate of https://github.com/copier-org/copier/issues/184, feel free to continue the conversation there.


Dead link in "File system sharing (osxfs)"

File: docker-for-mac/osxfs.md
Dead link: #preferences.

@tlxtellef - Very sorry for the delayed response on this - this link appears to be working. Please open another issue if you continue to experience a problem. Thank you!


When will EF7 for SQLite be Updated?  RC1 has Build Issue.

On my old inactive post #4331 there is an issue with building release versions using EF7 RC1 for SQLite.  When the project is built for release any calls to the SQLite DB fail .  I tried the suggestions but they did not help.
I have several apps I am ready to release to Windows store but cannot because if I build the projects for release the apps crash when run.  Building the apps for Debug make them run crazy slow on the phone. I was hoping I wasn't going to have to redo my entire DB approach and leave SQLite.

RC2 will be coming out as soon as everything is stabilized on the new .NET CLI. We don't have an exact date on that yet because the work is still ongoing. RC2 takes a vastly different approach to running under .NET Native - thanks to some improvements the .NET Native team made - so things should be a lot smoother in terms of release/,NET Native/publishing to store.


Improve SoundDidFinishPlayingNotification by including completion success in notification

Modules wanting to be notified about the completion of an AudioPlayer playing a sound have two options:

Provide completion handler, but only be notified about whether play completed successfully
Register for NotificationCenter, but only be notified about the player that completed

This creates an issue when the module wants to do something specifically based on the completion success while not knowing which of potentially multiple AudioPlayers is sending the message.
Fixing #1 requires changing the method signature. While I would encourage this to be implemented as it makes the completion handler more powerful, it will break the current API.
Fixing #2 is easy and non-breaking. I have implemented this here by adding a userInfo dictionary to the Notification including the boolean completion success.

I have also made a few small code changes to the init methods to use best Swift practices.

Updated CHANGELOG as requested.

Travis is failing due to Swiftlint, which doesn't make sense (I checked it locally), will merge anyway. Thanks for your PR!


Add contributing.md file

Without a CONTRIBUTING.md file, potential contributors may not know how to contribute effectively. This could lead to inconsistencies in contributions and confusion among contributors.
Missing Code of Conduct:

Thankyou for raising the issues @Nikhil0-3 but this issues already exists, refer #28 and #47


Installing py-mmd-tools with pip is currently broken

A misconfiguration in pyproject.toml was not caught in #284 and have left the package unable to be installed. A patch is included in #298 .

Closed by #298


Feature: load codesnippets from Github for sample codes

In language tabs, I would like to load code snippets for each tab from Github rather than hard-coding it in markdown file. In markdown file, I would specify the Github Url for each language. This would help me save duplication content and single file as source of truth.

At the linked issue I suggested a markdown pre-processor to pull in the necessary files. If a Ruby solution is required it looks as if guidedown or similar would do the job.

Yea, I don't think this is something we'd implement right in the repo, but it's also exactly the kind of thing a custom helper method in your config.rb could help with. Just rename your index.html.md to index.html.md.erb, and you should be good to go to use helper methods.


[glTF] Wrong object mesh when two objects have a same name without an import hint

Tested versions
v4.3.beta1.official [a4f2ea91a]
System information
Godot v4.3.beta1 - macOS 14.4.1 - Vulkan (Forward+) - integrated Apple M1 Max - Apple M1 Max (10 Threads)
Issue description
Having two objects named Cube and Cube-col breaks the import:
https://github.com/godotengine/godot/assets/60579014/d8b08dc7-7557-4d63-99b9-cdd2f5b5094a
Steps to reproduce

Make a blend file with a Cube in it
Export gltf and add it to scene in godot
Change the cube geometry, and add a -col hint to it to make it Cube-col. Export and reimport in godot
Add a new cube. It will get a name Cube
When you reimport the scene in godot you'll see that both cubes are using the mesh of the first cube

Minimal reproduction project (MRP)
MRP_gltf_duplicate.zip

@AThousandShips In #72483 @rburing says that -col suffix specifically isn't removed if it would result in a name collision which I can confirm:

So it seems like this is a separate issue because this one isn't about a name collision but about broken import despite there not being any name collision after import is done (And also #72483 doesn't have any mention of wrong meshes being displayed in the viewport)
Also more info:
I just noticed that restarting the editor fixes this issue (even though clicking "Reimport" on gltf doesn't), which makes the MRP that I attached useless, you can only reproduce this by following the steps


Upgrade object-path to version 0.11.5 or later

See the "object path" Dependabot alert in https://github.com/britishredcrosssociety/nesl-operating-manual/network/alerts
Recommended remediation: upgrade object-path to version 0.11.5 or later, like so
"dependencies": {   "object-path": ">=0.11.5" }
or
"devDependencies": {   "object-path": ">=0.11.5" }

This codebase is not in active use


Remove name-based width logic from date-input component

In https://github.com/alphagov/govuk-frontend/pull/857/files#diff-e94394b2ac1d4f73991af98e4fa34fa3L32 we removed styling which made the year field 4 characters wide, but was coupled to the field's name.
However, to avoid making this a breaking release, we added logic to automatically add the width classes based on the name.
We are now removing this behaviour. Instead, users need pass explicit classes for each field in the items object for the desired width of the input field:
{{ govukDateInput({
  "id": "dob",
  "name": "dob",
  "fieldset": {
    "legend": {
      "text": "What is your date of birth?"
    }
  },
  "items": [
    {
      "name": "day",
      "classes": "govuk-input--width-2"
    },
    {
      "name": "month",
      "classes": "govuk-input--width-2"
    },
    {
      "name": "year",
      "classes": "govuk-input--width-4"
    }
  ]
}) }}

Fixes: #905
Trello ticket: https://trello.com/c/xTcZVm2O/1391-1-remove-name-based-width-logic-from-date-input-component

This is a tricky problem. Our pattern for dates is to have short fields for day and month, and then a longer field for year.
I understand and agree with this change, it's just a shame that users have to add classes manually to get the standard component. It would be great if they didn't need to do that.

Rebased and merged in @igloosi's absence.


Fix Notification Popup Bar

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always
frustrated when [...]
Fix Popup Notification i.e. Idle , Loot Drop
Describe the solution you'd like
A clear and concise description of what you want to happen.
Add Color and Highlight to Names , Drops

Can you please be more clear with this request? As-is, I don't have a good idea of what you're requesting.


[Update Request]: "ThePBone.GalaxyBudsClient" - v4.6.0

What type of update are you requesting?
A new version of an existing package
Current Package Identifier
ThePBone.GalaxyBudsClient
Package Version
v4.6.0
Please describe the changes you would like to see
There is a new release available v4.6.0. Winget distributes a half year old binary.
The developer also changed his GitHub username from ThePBone to timschneeb. I updated the manifest and the PR I opened is totally ready to merge since I tested everything and it works.

https://github.com/microsoft/winget-pkgs/pull/153569#issuecomment-2110726042


Update index.md

Pull Request Test Coverage Report for Build 4722

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 92.865%

Totals

Change from base Build 4719:
0.0%

Covered Lines:
1067

Relevant Lines:
1103

💛  - Coveralls

not sure why CI failed on node 14 even though this PR only contains one single change to docs folder

I've restarted the build. I think there is some undeterministic test in our setup that's causing this. There are other PRs with the build passing on two node versions (random) and failing on one. I've restarted those as well


Fix crashes on v1.19.4

The changes actually fix the migration test
Can you try run that locally and see?


How would I go about updating user data from the API?

I have an IdentityServer4+ASP.NET Identity Auth Server working with a JS client & an ASP.NET Core API. The JS client is able to use the Auth Server to register/login/logout and get the access token that would allow access to the API; a very similar setup to the 'Using ASP.NET Identity' quickstart in the docs.
Now, how would I allow the API to edit user data (eg. update profile image, 'favorites', prefs, etc.)? Currently the API can only access the user data through the token and doesn't communicate with the Auth Server (which is the only server that has access to my users database). Is there some built-in mechanic for this that I'm missing or do I have to grant my API write access to my users database?
I apologize for posting a question in the issues section but this is the only active place I could think of.

You need to provide your own API to write to your user database.

I want to edit a user details. how should i create a edit action in Account controller?


Notification builder : more explicit logs in case of sending failure.

Logging the effective recipients in addition to the original ones could
be useful in development or staging mode.
Logging the sending address (from) could be useful at any time.
The structure I suggest in this PR can be discussed, but I didn't want to catch any MessagingException in the catch (MailException e) block (which would have a poor readability).
Produced logs sample : send-notification-log-improvement.txt

Added replyTo and deleted message != null test.


sometimes bs58 not decoding correctly

Hello,
hashcode is getting generated from murmurhash with uuid v4 when i trying to encode and decode sometimes it is working correctly and sometime working incorrectly
please follow the below code and correct and incorrect results
const bs58 = require('bs58')
//"bs58": "^4.0.1",
const murmur = require('murmurhash')
//"murmurhash": "0.0.2",
const uuidv4= require('uuid/v4')
//"uuid": "^3.3.3"
const hashcodeBefore = murmur(uuidv4())
console.log(hash code =>${hashcodeBefore})
const hexValueBeforeEncoding = hashcodeBefore.toString(16)
console.log(hex value => ${hexValueBeforeEncoding})
// encode
const encode = bs58.encode(Buffer.from(hexValueBeforeEncoding,'hex'))
console.log(encode values is => ${encode})
//decode
const hexValueAfterDecoding = bs58.decode(encode).toString('hex');
console.log(decode to hex value => ${hexValueAfterDecoding})
const hashcodeAfter = parseInt(hexValueAfterDecoding,16)
console.log(hash code => ${hashcodeAfter})
correct results
1)
hash code is<PHONE_NUMBER>
hex value => abcfdaec
encode values is => 5PihXy
decode to hex value => abcfdaec
hash code =><PHONE_NUMBER>
2)
hash code =>842513527
hex value => 3237bc77
encode values is => 2HT6vJ
decode to hex value => 3237bc77
hash code => 842513527

hash code =>4034442918
hex value => f078b6a6
encode values is => 79WaKf
decode to hex value => f078b6a6
hash code =><PHONE_NUMBER>

hash code =>3752366994
hex value => dfa89392
encode values is => 6iaryf
decode to hex value => dfa89392
hash code =><PHONE_NUMBER>
incorrect result
1)
hash code =>205424363
hex value => c3e86eb
encode values is => 28obT
decode to hex value => c3e86e ( character b is missing here)
hash code => 12839022 (incorrect hashcode value generated)

hash code =>70739630
hex value => 43766ae
encode values is => PfH3
decode to hex value => 43766a (character e is missing here)
hash code => 4421226 ( incorrect hascode value generated)

hash code =>66211204
hex value => 3f24d84
encode values is => ND9H
decode to hex value => 3f24d8 (character 4 is missing here)
hash code => 4138200 ( incorrect hascode value generated)

hash code =>189365373
hex value => b497c7d
encode values is => 23fEW
decode to hex value => b497c7 (character d is missing here)
hash code => 11835335 ( incorrect hascode value generated)
is there any thing i am doing wrong in this code please check and let me know and thanks in advance

It's happened because you create buffer not from aligned hex:
Buffer.from('c3e86eb', 'hex')
<Buffer c3 e8 6e>

b is missed in buffer.
You can solve it with String#padStart
const hex = 'c3e86eb'
Buffer.from(hex.padStart(hex.length + hex.length % 2, '0'), 'hex')
<Buffer 0c 3e 86 eb>


Status Prefab is missing

You have to sync the Git submodules.

I'm closing this ticket. Please feel free to reopen for further questions.


was there ever a fix for this xteve error EC: 4007

error: The process cannot access the file because it is being used by another process. (Old temporary buffer file could not be deleted) - EC: 4007
i believe its casing my streams to get pixelated and then freezes if i stop the stream and restart the same stream. i dont understand why xteve is not able to delete temporary files that it created in the firat place. however. if i close xteve the files get deleted but it happens agin once i play a stream and that error keeps popping up in th exteve console for every ts file it creates and tries to delete.

I assume your using the buffer to remux with a temp folder xTeVe has permissions to?

It depends on which format you use...Docker/Windows/Linux?

It should say in the Web UI.....

Windows 10
On Sat, 18 Feb 2023, 1:53 am Fith, @.***> wrote:

It should say in the Web UI.....
—
Reply to this email directly, view it on GitHub
https://github.com/xteve-project/xTeVe/issues/355#issuecomment-1435502370,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AODRSPN6PTB23K3UV4YYC4TWYBWWVANCNFSM6AAAAAAU64KOGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

see it doesnt say the default path after ive chaneg the temp path

Just leave it blank and xTeVe will use the default path.

awesome. thanks. i havent seen the error pop up now that ive changed it to the default.

i was wrong. im still getting the error with "C:\Users\User0\AppData\Local\Temp\xteve"*

User0 is what you called your Windows user?

Yes
On Mon, 20 Feb 2023, 3:24 pm Fith, @.***> wrote:

User0 is what you called your Windows user?
—
Reply to this email directly, view it on GitHub
https://github.com/xteve-project/xTeVe/issues/355#issuecomment-1437512900,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AODRSPJ7TPGOCYGLJOXSYATWYPHJHANCNFSM6AAAAAAU64KOGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

So I assume you are running xTeVe by clicking on the .exe from your desktop and it opens in a Terminal?

I made a batch file to run the exe and changes the port number. Amd it's in
a different folder, not my desktop or downloads
On Wed, 22 Feb 2023, 1:53 am Fith, @.***> wrote:

So I assume you are running xTeVe by clicking on the .exe from your
desktop and it opens in a Terminal?
—
Reply to this email directly, view it on GitHub
https://github.com/xteve-project/xTeVe/issues/355#issuecomment-1439521497,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AODRSPJ25HQXI227C5Y43VDWYWZW7ANCNFSM6AAAAAAU64KOGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

What runs the batch file?

Also, look at what client is playing the streams. Could be that the client is not letting go and closing the connection properly.

I know from experience, Plex is a train wreck of issues.

I use emby as a client and server
On Wed, 22 Feb 2023, 2:13 am Fith, @.***> wrote:

I know from experience, Plex is a train wreck of issues.
—
Reply to this email directly, view it on GitHub
https://github.com/xteve-project/xTeVe/issues/355#issuecomment-1439536242,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AODRSPLOQYOBCMMRA6VIY2DWYW4C7ANCNFSM6AAAAAAU64KOGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

I would reach out to Luke at Emby with your logs and see why client is not disconnecting.


Update CHANGELOG.md

Update to reflect merged if statements in src/main/java/org/trebol/jpa/services/conversion/PeopleConverterJpaServiceImpl.java

Hi @JellyBoonz, welcome and thank you for your interest!
I'm sorry though, but you have to add this commit to your parent PR for this matter, #195
To do that, just follow three steps

checkout (switch to) your patch-1 branch
merge patch-2 into it
push patch-1 again

Then, the GitHub PR page will update itself.
Also, please bear in mind the review & comment that NyorJa left for you in that PR as well.
See you on the other side! ✌


Member "mockCall" not found or not visible after argument-dependent lookup in contract Vm. solidity(9582)

Component
Forge
Have you ensured that all of these are up to date?

[X] Foundry
[X] Foundryup

What version of Foundry are you on?
forge 0.2.0 (7b45265 2023-11-21T00:18:30.267248000Z)
What command(s) is the bug in?
forge test -vvvv
Operating System
macOS (Apple Silicon)
Describe the bug
I am trying to use the mockCall function in a test file as described in the book, but the compiler doesn't seem to recognise the exported (external) mockCall from the imported forge-std/Test.sol contract (Vm).
All the other vm's functions in the snippet below work with no issues.
Code snippet:
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.17;

import "forge-std/Test.sol";

contract SafePayoutsModuleTest is Test {
   ...

    function test_executePayouts_OperatorIsAllowed() public {
        vm.deal(address(safeAccount), 10 ether);
        assertEq(address(safeAccount).balance, 10 ether);

        vm.startPrank(address(this));
        payoutsModule.addPayout(alice, 1 ether);
        payoutsModule.addPayout(bob, 2 ether);
        payoutsModule.addPayout(charlie, 3 ether);
        vm.stopPrank();

        vm.mockCall(
            safeAccount,
            0,
            abi.encodeWithSelector(safeAccount.execTransactionFromModule.selector, address(alice), 1 ether, "", 0),
            abi.encode(true)
        );

        ...

   }
}

Error log:
➜ forge test -vvvv
[⠰] Compiling...
[⠆] Compiling 12 files with 0.8.23
[⠰] Solc 0.8.23 finished in 886.64ms
Error:
Compiler run failed:
Error (9582): Member "mockCall" not found or not visible after argument-dependent lookup in contract Vm.
   --> test/SafePayoutsModule.t.sol:126:9:
    |
126 |         vm.mockCall(
    |         ^^^^^^^^^^^

What type is safeAccount? I see you used address(safeAccount) above so can you try that?