Superset redirects to http from https

We have deployed superset to an EKS cluster. We want TLS terminated at the Ingress controller.
This is how I set up the entire setup.
GSLB Vip → Nginx ingress controller → http → ingress → service → pod
When we enter the URL/login/ on the browser, it gets redirected to login again after we provide the credentials. On a new tab if I then open /superset/welcome it works just fine. We assume the problem is because Superset redirects initially to http from https, and the login in inaccessible, but I could be wrong
When we curl the WIP, this is the response
curl -ivk https://superset-aws-ssl.g.xx.com
*   Trying 10.20.xx.66...
* TCP_NODELAY set
* Connected to superset-aws-ssl.g.xx.com (10.20.xx.66) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / xxx-xxx-xxx-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=superset-aws-ssl.g.xx.com; OU=xx; O=YY Inc.; ST=CC; C=US
*  start date: Oct 13 20:01:12 2021 GMT
*  expire date: Nov 12 20:01:11 2023 GMT
*  issuer: CN=xx CA 1; OU=Certification Authority; O=xx; C=US
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: superset-aws-ssl.g.xx.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 302 FOUND
HTTP/1.1 302 FOUND
< Server: nginx/1.17.8
Server: nginx/1.17.8
< Date: Fri, 15 Oct 2021 01:42:21 GMT
Date: Fri, 15 Oct 2021 01:42:21 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Content-Length: 243
Content-Length: 243
< Connection: keep-alive
Connection: keep-alive
< Location: http://superset-aws-ssl.g.xx.com/superset/welcome/
Location: http://superset-aws-ssl.g.xx.com/superset/welcome/

<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>

Redirecting...

In the above curl, we see that the location is getting redirected to HTTP.
This is the ingress controller yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Forwarded-Proto: $scheme";
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:vvvv:certificate/xx-x-x-x-xxx
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: ingress-nginx
  namespace: ingress-nginx
  resourceVersion: "6835154"
  selfLink: /api/v1/namespaces/ingress-nginx/services/ingress-nginx
  uid: xx-xx-xx-xx-xxxxx
spec:
  clusterIP: xx.xxx.xxx.xxx
  externalTrafficPolicy: Local
  healthCheckNodePort: 30599
  ports:
  - name: http
    nodePort: 31840
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 30330
    port: 443
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer

This is the ingress on top of the superset service:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Forwarded-Proto: https";
  name: superset-ingress
  namespace: superset-ssl-v2
spec:
  rules:
  - host: https://superset-aws-ssl.g.xx.com
    http:
      paths:
      - path: /(.*)
        pathType: Prefix
        backend:
          serviceName: superset
          servicePort: 8080

When we port forward the service, it works just fine.

I am also experiencing the same issue

@cintoSunny There is one mistake I see in your annotations. Actually the doc mentions to add a request header while the more_set_headers add a response header. Below configurations changes worked for me:

Add request headers using custom configmap as per nginx ingress documentation:

apiVersion: v1
data:
  X-Forwarded-Proto: "https"
kind: ConfigMap
metadata:
  name: custom-headers
  namespace: ingress-nginx

Refer to the above custom configmap in the configmap passed to nginx-ingress command (--configmap=$(POD_NAMESPACE)/ingress-nginx-controller). proxy-set-headers is the important part here which sets it as request header:

apiVersion: v1
data:
  proxy-set-headers: "ingress-nginx/custom-headers"
kind: ConfigMap
metadata:
  name: private-nginx-controller
  namespace: nginx

Add below flags to superset config:

ENABLE_PROXY_FIX = True
PROXY_FIX_CONFIG = {"x_for": 1, "x_proto": 1, "x_host": 1, "x_port": 0, "x_prefix": 1}

I'm setting "x_port": 0 in PROXY_FIX_CONFIG without which I was getting Location as https://superset.example.com:80/. This combination of 3 changes fixed it form me.


Update to latest core commit before component move

Signed-off-by: Bogdan Drutu<EMAIL_ADDRESS>

Please check lint errors


playwright: init at 1.21.0

Description of changes
https://github.com/microsoft/playwright-python
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Thanks @ckiee for your feedback :)

1.22.0 has since been released, and so this PR now inits at that version

@ckiee I've just pushed a change so it'll use nix's nodejs

[...] but the playwright CLI tried to download browser binaries and run them which doesn't work on NixOS

@ckiee This is how I'm currently doing it:
{
  playwright-browsers = runCommand "playwright-browsers" {
    outputHashMode = "recursive";
    outputHashAlgo = "sha256";
    outputHash = {
      x86_64-linux = "sha256-N7kFqB0qvG1g08OP9EestNC6h1f7ICPAj8GK3pQs2FA=";
      x86_64-darwin = "sha256-/4kPKyK5rrMywWwi8owDfK43wtp3FnfdurdRODmDGfA=";
    }.${stdenv.hostPlatform.system} or (throw "Unsupported system");
  } ''
    export PLAYWRIGHT_BROWSERS_PATH=$out
    ${python.pkgs.playwright}/bin/playwright install
  '';
}

Here, python is python310 with an override for the Playwright package from this PR. Use it by setting the PLAYWRIGHT_BROWSERS_PATH environment variable to the above derivation.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/differences-in-evaulation-with-fixed-output-dependencies-under-darwin-and-linux/20592/1

@yrd @ckiee thank you both for reviewing this PR. As @ckiee has said in one of my other PRs, I should really be testing with NixOS, as using darwin lots of system libraries already exist that I am relying on. However, even with playwright distributing browsers itself, there are system libraries that are still needed when using NixOS.
This PR needs more work (the FHS stuff especially that @ckiee mentioned above, and determining which specific system libs are needed), however I'm not able to commit the time needed for this specific PR as I'm not using playwright anymore.
I'm going to close this PR now, but if anyone else would like to continue it I would be happy for you to use any piece of this existing PR if it can help it any way.


Setup layout for thread and comment pages

This PR is to be merged after #34.
Changes:

Create a thread-list component to display a list of threads . Show timestamp, primaryTag and topic of each thread.
Created a thread-detail component to display detaled information(comments, statistics) of a single thread. Hence, with routing enabled in #34, user can click on a single thread in thread-list and navigate into each thread.

Screenshots:

Show a list of threads on clicking search.

When clicking on each thread, jump to thread detail page with a list of comments and tag statistics.

Changes proposed in this PR has a large overlap with another PR #36. Problems in this PR are also addressed in #36.
I am going to close this PR.


First time running kops_cluster_updater fails

Hi again :)
Goal
Trying to spin up a simple 100% spot autoscaling kops cluster from scratch.
Problem
kops_cluster_updater never creates succesfully, not in the initial run, and not in subsequent runs when everything except the updater has been successfully created.
I also tried a terraform destroy -target kops_cluster.cluster, manually wiping the S3 bucket, and re-running, to no avail.
Error
Have you seen the first error below before? It looks like something to do with networking, but that sounds weird to me, as I would assume there can't be any networking (errors) before any EC2 instances have been created.
As I understand it (please correct me though), the kops_cluster_updater resource is responsible for actually spinning up EC2 instances for the master instance group (min 1 max 1). At this point in time, kops_instance_group resources are successfully created (and the S3 bucket has been populated), but I don't yet see a new EC2 instance in AWS UI for the master instance group.
The second error occurs when I try to kubectl apply a manifest to the new kops cluster. Probably this kubectl resource (the very last one in kops.tf) needs a depends_on = [kops_cluster_updater.updater]? And even then, how would api.kops-cluster-0.rancher.delange.dev start resolving if I didn't set up any DNS?
kops.tf
# for kops cluster, one master per availability zone is needed ref https://kops.sigs.k8s.io/tutorial/working-with-instancegroups/
# this data-source assumes there is one subnet per availability zone. if there are multiple subnets, this is a FIXME
data "aws_subnets" "all" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.default.id]
  }

  # FIXME for now only running in one availability zone. delete this filter to get all subnets for the vpc
  filter {
    name   = "availability-zone"
    values = ["eu-central-1a"]
  }
}

data "aws_subnet" "all" {
  # for_each           = data.aws_subnet_ids.all.ids
  for_each           = toset(data.aws_subnets.all.ids)
  id                 = each.key
}

# defining local because it needs to be used for both the provider and the resource (provider can't read it from resource, would be cyclic)
locals {
  kops_cluster_name       = "kops-cluster-0.${var.rancher_server_dns}"
}

resource "aws_s3_bucket" "kops_state_bucket" {
  bucket = local.kops_cluster_name
  acl    = "private"
}

provider "kops" {
  state_store = "s3://${local.kops_cluster_name}"
}

resource "kops_cluster" "cluster" {
  name               = local.kops_cluster_name
  admin_ssh_key      = tls_private_key.global_key.public_key_openssh  # FIXME that's the key of the machine that originally ran terraform apply
  cloud_provider     = "aws"
  kubernetes_version = "v1.21.7"  #  pair with kops provider version ref https://kops.sigs.k8s.io/welcome/releases/
  dns_zone           = data.aws_route53_zone.this.name
  network_id         = data.aws_vpc.default.id

  iam {
    allow_container_registry = true
  }

  networking {
    calico {}
  }

  topology {
    masters = "private"
    nodes   = "private"
    dns {
      type = "Private"
    }
  }

  # private subnets, one per availability zone
  dynamic "subnet" {
    for_each    = data.aws_subnet.all
    content {
      name        = subnet.value.id  # using id as name because the name field is empty for default subnets, this is also used in instance groups below
      type        = "Private"
      provider_id = subnet.value.id
      zone        = subnet.value.availability_zone
    }
  }

  # etcd clusters
  etcd_cluster {
    name = "main"
    dynamic "member" {
      for_each = data.aws_subnet.all
      content {
        # matching kops_instance_group.masters
        name           = "master-${member.value.availability_zone}"
        instance_group = "master-${member.value.availability_zone}"
      }
    }
  }
  etcd_cluster {
    name = "events"
    dynamic "member" {
      for_each = data.aws_subnet.all
      content {
        # matching kops_instance_group.masters
        name           = "master-${member.value.availability_zone}"
        instance_group = "master-${member.value.availability_zone}"
      }
    }
  }
}

# master instance group
resource "kops_instance_group" "masters" {
  # one master per availability zone, and one subnet per availability zone -> one master per subnet (ref https://kops.sigs.k8s.io/tutorial/working-with-instancegroups/)
  for_each     = data.aws_subnet.all
  cluster_name = kops_cluster.cluster.name
  name         = "master-${each.value.availability_zone}"
  role         = "Master"
  min_size     = 1
  max_size     = 1
  machine_type = "t3a.small"  # FIXME might need to go back to t3a.medium
  subnets      = [each.value.id]  # can also use each.value.availability_zone
}

# worker instance group
resource "kops_instance_group" "nodes" {
  for_each     = data.aws_subnet.all  # one node instance group per availability zone
  cluster_name = kops_cluster.cluster.name
  name         = "node-${each.value.availability_zone}"
  role         = "Node"
  min_size     = 0  # let's hope this works ref https://github.com/kubernetes/autoscaler/issues/2418#issuecomment-541369677
  max_size     = 2
  machine_type = "t3a.small"  # why is this required when using mixed_instances_policy, will it be overwritten or still somehow used?
  subnets      = [each.value.id]  # can also use each.value.availability_zone
  mixed_instances_policy {
    instances = ["t3a.small", "t3.small"]
    # ref https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html#allocation-strategies
    on_demand_allocation_strategy = "lowest-price"
    # ref https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html#spot-best-practices
    spot_allocation_strategy = "lowest-price"
    spot_instance_pools = 10  # larger than necessary: use all instance types
    # ref https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html#instances-distribution
    # different syntax https://github.com/eddycharly/terraform-provider-kops/issues/405
    on_demand_base { value = 0 }
    on_demand_above_base { value = 0 }
  }
  # ref https://artifacthub.io/packages/helm/cluster-autoscaler/cluster-autoscaler#aws---using-auto-discovery-of-tagged-instance-groups
  cloud_labels = {
    "k8s.io/cluster-autoscaler/enabled" = ""
    "k8s.io/cluster-autoscaler/${kops_cluster.cluster.name}" = ""
  }
}

resource "kops_cluster_updater" "updater" {
  cluster_name        = kops_cluster.cluster.name

  keepers = merge(
    {cluster  = kops_cluster.cluster.revision},
    {for g in kops_instance_group.masters : g.name => g.revision}
  )

  rolling_update {
    skip                = false
    fail_on_drain_error = true
    fail_on_validate    = true
    validate_count      = 1
  }

  validate {
    skip = false
  }

  # ensures rolling update happens after the cluster and instance groups are up to date
  depends_on   = [kops_cluster.cluster, kops_instance_group.masters]
}

####
# Connect the kops cluster to rancher
####

# rancher-common module (rancher.tf) creates a new cluster entry and sets up the cluster autoscaler helm chart (helm.tf)

# GET the registration token (and so the kubectl manifest) ref https://github.com/rancher/terraform-provider-rancher2/issues/795#issuecomment-965116629
data "http" "rancher_registration_manifest" {
  # url = rancher2_cluster.kops_cluster.cluster_registration_token[0].manifest_url
  url = module.rancher_common.workload_cluster_manifest_url
  depends_on = [aws_route53_record.rancher_a_record]  # needs to be reachable
}

# get the kube config of the kops cluster for running kubectl
data "kops_kube_config" "kube_config" {
  cluster_name = kops_cluster.cluster.name
  # ensure the cluster is up
  depends_on   = [kops_cluster.cluster, kops_instance_group.masters]
}

# set up the kubectl provider with the kops config
provider "kubectl" {
  host                   = data.kops_kube_config.kube_config.server
  username               = data.kops_kube_config.kube_config.kube_user
  password               = data.kops_kube_config.kube_config.kube_password
  client_certificate     = data.kops_kube_config.kube_config.client_cert
  client_key             = data.kops_kube_config.kube_config.client_key
  cluster_ca_certificate = data.kops_kube_config.kube_config.ca_cert
}

# apply the manifest to import the kops cluster into rancher
resource "kubectl_manifest" "rancher_import_manifest" {
  yaml_body = data.http.rancher_registration_manifest.body
}

kops_cluster_updater.updater: Still creating... [10m20s elapsed]
╷
│ Error: error running tasks: deadline exceeded executing task ClassicLoadBalancer/api.kops-cluster-0.rancher.delange.dev. Example error: Field is required: Subnets
│
│   with kops_cluster_updater.updater,
│   on workload.tf line 138, in resource "kops_cluster_updater" "updater":
│  138: resource "kops_cluster_updater" "updater" {
│
╵
╷
│ Error: proxy-clusterrole-kubeapiserver failed to create kubernetes rest client for update of resource: Get "https://api.kops-cluster-0.rancher.delange.dev/api?timeout=32s": dial tcp: lookup api.kops-cluster-0.rancher.delange.dev on <IP_ADDRESS>:53: no such host
│
│   with kubectl_manifest.rancher_import_manifest,
│   on workload.tf line 192, in resource "kubectl_manifest" "rancher_import_manifest":
│  192: resource "kubectl_manifest" "rancher_import_manifest" {
│
╵

Acting upon Field is required: Subnets from the error above, I added:
resource "kops_cluster" "cluster" {
  ...
  # omitting api block will throw error when creating cluster_updater resource
  api {
    # dns {}
    load_balancer {
      class = "Classic"
      type = "Public"
      dynamic "subnets" {
        for_each    = data.aws_subnet.all
        content {
          name        = subnets.value.id
        }
      }
    }
  }
}

which results in a warning and a new error:
$ TF_LOG=INFO terraform apply -auto-approve
2021-11-29T12:56:26.042+0200 [INFO]  Starting apply for kops_cluster_updater.updater
2021-11-29T12:56:26.304+0200 [WARN]  Provider "provider[\"registry.terraform.io/eddycharly/kops\"]" produced an unexpected new value for kops_instance_group.nodes["subnet-87b83aed"], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .root_volume_encryption_key: was null, but now cty.StringVal("")
      - .root_volume_optimization: was null, but now cty.False
      - .root_volume_size: was null, but now cty.NumberIntVal(0)
      - .cpu_credits: was null, but now cty.StringVal("")
      - [...]
[...]
kops_cluster_updater.updater: Still creating... [15m21s elapsed]
2021-11-29T09:46:09.906+0200 [INFO]  provider.terraform-provider-kops_v1.21.2-alpha.2 [...] unexpected error during validation: error listing nodes:  Get "https://api-kops-cluster-0-[...]

As both the kops-created classic ELB's status, and the new kops master EC2 instance's health checks look good, I'm suspecting the security group created by kops to be the culprit. Probably because I'm using the availability zone's default VPC and its subnet. Maybe when letting kops create VPC (isn't that default?), also the security group would be managed by kops. But now it's my responsibility, as I'm passing the AZ's default VPC in kops_cluster.cluster.network_id.
Adding a security_group_override = aws_security_group.my_custom_allowall_sg.idin the newapi.load_balancer` block above (and to both master & node instance groups), and will try to see if that solves the issue with ELB.

I have the exact same issue. It timeouts on terraform apply and for what I've seen, I needed to change the topology to public and use only public subnets, but with no success.

Update on my last paragraph:
The EC2 instances and the ELB are now successfully created, but the ELB shows OutOfService error, meaning the ping on 443 is failing. Definitely a networking issue. The error now slightly changed with empty string instead of api URL:
error listing nodes: an error on the server ("") has prevented the request from succeeding (get nodes)

Will try adding a NAT gateway which I think is required for private subnets. Ref https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html

Regarding OutOfService, I think it's because they're not public. Make sure to make them sit in a public subnet with topology set to public.
I have tried to use kopsctl for a raw cluster, without Terraform, and it seems like you need to add the Route53 nameservers to your domain provider. The kops Terraform Provider creates the Route53 zone for you but inspecting the zone in the Route53 dashboard you'll see 4 nameservers, which need to be added to your domain. Just wait for them to propagate and retry running the terraform.

Hello, sorry for the late reply here.
I think your issue is because you're trying to create a cluster with a private topology in the default vpc.
I don't remember how the default VPC is setup but i'm not sure at all it would work.
To create a cluster with a private topology you will need:

private subnets with nat gateway to connect to internet
public subnets with direct access to internet through an internet gateway

In addition, i think you will need to tag your subnets to enable discovery:

public subnets tags:          SubnetType: Utility
          kubernetes.io/cluster/<cluster name>: shared
          kubernetes.io/role/elb: 1

private subnets tags:          SubnetType: Private
          kubernetes.io/cluster/<cluster name>: shared
          kubernetes.io/role/internal-elb: 1

Moreover, if you create a cluster with private topology, you need to be in the private network to access the api, did you set up a vpn or something ?
Also, you shouldn't need to specify the subnets by hand.
Of course, this also applies to dns, if you use a private zone, you need to make sure you can resolve the host names.
Finally, a kOps cluster cannot start it there are only master nodes, you seem to have your worker nodes min set to 0:
  min_size     = 0  # let's hope this works ref https://github.com/kubernetes/autoscaler/issues/2418#issuecomment-541369677

Let me know if my comments make sense and how i can help.

One more thing:
resource "kops_instance_group" "nodes" {
  // ...
  cluster_name = kops_cluster.cluster.id
  // ...

You should change kops_cluster.cluster.name by kops_cluster.cluster.id to make sure the creation order is correct.

If you need inspiration we have a couple of modules at https://registry.terraform.io/modules/opzkit/k8s/aws/latest and https://registry.terraform.io/modules/opzkit/k8s-network/aws/latest which we have successfully used ourselves and for a couple of clients.

@ddelange can we close the issue ? You can always reopen it or create another one if necessary

Sure, closing! Thanks for the elaborate responses 👍


Update tinylicious code for latest server changes.

Tinylicious was consuming pre-release dependencies (#3673) but it was not built after some recent changes to 0.1014.0 (#3818 and #3998 ). This caused the failure of recent release bump (#4181 ). This PR should fix the build failure.

FYI @curtisman and @ksbrar


haskell infrastructure: moving forward without peti

@peti posted a thread on Discourse about stopping work on the Haskell infrastructure here in Nixpkgs.
peti wrote a large part of the Haskell infrastructure here in Nixpkgs, and has done a large part of the work in maintaining it over the years.  I think a lot of Haskellers who use Nix have directly benefited from all the work peti has done.  In my experience, Haskell is one of the nicest languages to use with Nix, and this is in no small part due to peti.
It is really going to hurt having peti step away from Nixpkgs.  But we'll need to find a way to move forward with the Haskell infrastructure here in Nixpkgs.
In particular, we need people to step in and do the tasks peti has been doing for the past couple years.
As far as I know, here are the main tasks peti is doing that will need to be taken over:

Open a PR every week for the haskell-updates branch that targets master.  Here is an example of one of these PRs.
Sometime during the week, update the Stackage Nightly target in a commit on haskell-updates.  Here is an example of one of these commits.
Once a week, make sure to fix up anything on the haskell-updates branch that is not evaluating correctly, and merge the above PR.  This is somewhat of a manual task, but I think peti relies on Hydra a lot during this.  peti has a Twitch channel where he often does this live: https://www.twitch.tv/peti343.  (As an aside, do past videos disappear from twitch at some point?  @peti, I thought you had uploaded some videos to YouTube in order to make sure that they are always available, but I can't find them anymore with a quick search. edit: Yes, @sternenseemann confirmed that they are deleted.)
Mark Haskell packages as broken when they no longer build on haskell-updates.  I think peti relies on Hydra for this as well.
Make sure the hackage-updates cron job is running.  This generates the daily Hackage updates and commits them to Nixpkgs on the haskell-updates branch.  Here is an example of a commit that was generated by this.
Review any PRs to hackage2nix (and the related cabal2nix).
Make sure that the haskell-updates jobset is working on Hydra: https://hydra.nixos.org/jobset/nixpkgs/haskell-updates.  (I'm not actually sure what goes into this or how much work it really is.)

(If I've forgotten anything important, please feel free to edit this post and to this list.)
Aside from peti, the main committers that have been active with the haskell-updates branch have been me, @maralorn, and @sternenseemann .  I'm imagining that one of us would take over these tasks.  Or possibly split it up between us.
Other people that I think might be interested in the Haskell infrastructure here in Nixpkgs: @domenkozar, @Ericson2314, @jonringer, @bgamari, @roberth, @angerman, @hamishmack, @nh2

As an aside, I'd like to suggest that in this thread, we focus on figuring out who personally will be responsible for the above tasks peti used to perform.
It is tempting to also discuss various ways the haskell-updates workflow could be changed, but we should probably keep that to a separate thread.  If you'd like o discuss something that would be a significant change, please open up a separate thread (and post a link to it here).

Once a week, make sure to fix up anything on the haskell-updates branch that is not evaluating correctly, and merge the above PR.  This is somewhat of a manual task, but I think peti relies on Hydra a lot during this.  peti has a Twitch channel where he often does this live: https://www.twitch.tv/peti343.  (As an aside, do past videos disappear from twitch at some point?  @peti, I thought you had uploaded some videos to YouTube in order to make sure that they are always available, but I can't find them anymore with a quick search.)

IIRC, the recordings are deleted after a few weeks automatically.

Make sure the hackage-updates cron job is running.  This generates the daily Hackage updates and commits them to Nixpkgs on the haskell-updates branch.  Here is an example of a commit that was generated by this.

@peti has agreed to keep this up for a few more weeks, maybe we can migrate this to a github action in the mean time to make it more maintainable for us?

Review any PRs to hackage2nix (and the related cabal2nix).

I'd be happy to be more involved with this if it's fine with @peti.

Make sure that the haskell-updates jobset is working on Hydra: https://hydra.nixos.org/jobset/nixpkgs/haskell-updates.  (I'm not actually sure what goes into this or how much work it really is.)

I think it's manageable, but I'd be great if the current active haskell maintainers (@cdepillabout @maralorn, me) could be given access to the jobset plus some basic hydra privileges (restart-jobs, cancel-build could also be useful).

Aside from peti, the main committers that have been active with the haskell-updates branch have been me, @maralorn, and @sternenseemann . I'm imagining that one of us would take over these tasks. Or possibly split it up between us.

I don't think I have the time regularly enough to take over, but I'm happy to contribute to this. If we want to make the final fixup phase a more joint effort we maybe need to think about rescheduling what was the livestreams as 20:00 UTC+2 may line up for @maralorn and me well, it probably doesn't for you.

As an aside, I'd like to suggest that in this thread, we focus on figuring out who personally will be responsible for the above tasks peti used to perform.
It is tempting to also discuss various ways the haskell-updates workflow could be changed, but we should probably keep that to a separate thread. If you'd like o discuss something that would be a significant change, please open up a separate thread (and post a link to it here).

I agree to this sentiment: We should focus on keeping the show running as is for now. However I think we should organize it in a way that it doesn't hinge on a single person (to avoid a situation like this in the future).

A couple additional questions I have for @peti:

Would you be able to make me, @maralorn, and @sternenseemann committers in cabal2nix so that we could make any required changes to hackage2nix?
I know of https://github.com/peti/ci, and I think there is some relationship to Hdyra, but I've really understood the how the two work together?  Would you be able to give a quick explanation of what this repo is, and how it affects haskell-updates on Hydra?  Would it make sense for me, @maralorn, and @sternenseemann to have commit rights here too?

Here are my responses to @sternenseemann's message in https://github.com/NixOS/nixpkgs/issues/121140#issuecomment-829141128:

peti has agreed to keep [the hackage-updates cron script] up for a few more weeks, maybe we can migrate this to a github action in the mean time to make it more maintainable for us?

Oh, a GitHub Action is a really good idea.  Here's some documentation about scheduled actions: https://jasonet.co/posts/scheduled-actions/.  Implementing this should probably be split off to a separate issue.

Make sure that the haskell-updates jobset is working on Hydra

I think it's manageable, but I'd be great if the current active haskell maintainers (@cdepillabout @maralorn, me) could be given access to the jobset plus some basic hydra privileges (restart-jobs, cancel-build could also be useful).

Yes, I definitely agree with this.
However, I brought it up before and it didn't really lead anywhere: https://discourse.nixos.org/t/how-to-get-privileges-on-hydra-to-restart-builds/9757
I imagine it might be difficult for us to continue working on the haskell-updates branch if none of us have the ability to do anything on Hydra.
I'm not sure who to continue to ask about this though.  Maybe @domenkozar or @zimbatm?

I also wanted to post about how I feel moving forward.
Up until now, I've mostly been helping people who send PRs to the haskell-updates branch, but not actively involved in the stuff peti does (clean up the haskell-updates branch and merge it into master).  I would be completely happy not being involved with the work that goes into cleaning up and merging in the haskell-updates branch.  However, if no one else is going to do it, I would definitely take up the reigns.  I might want to switch to a twice-monthly merge schedule instead of a weekly merge schedule to reduce the workload though.

However I think we should organize it in a way that it doesn't hinge on a single person (to avoid a situation like this in the future).

I completely agree with this.  Ideally we'd like to find some workflow that doesn't completely hinge on a single person.  But for now, it seems like it might be easiest to have a single person who is ultimately responsible to get haskell-updates merged into master (along with everything that implies).

I think since cabal2nix belongs to the nix org, we already have commit rights there.
@cdepillabout I am sorry, but I feel that can‘t talk about the who without the how.
I personally am in the end phase of my thesis and am not able to take on new obligations in the next year.
I think petis workflow was petis workflow. Finding someone to be the new peti seems not like the perfect solution.
I have a lot of ideas for the future of haskellPackages but I agree, that we should postpone getting crazy with that. (That being said I would love it if some kind of task force came together and would discuss this more).
A suggestion I have, would be to break free of the weekly schedule. And make the cleanup more a joined effort. I think one way to lessen the churn would be to stop doing nightly hackage updates. I would do one stackage+hackage update, clean up the PR over a few days and merge it. nixpkgs master will be a few days more behind hackage. But that seems totally okay to me on a volunteer basis.
Also I think we should push quickly in the direction that all packages someone cares about have a maintainer, so that the person doing the PR only needs to do a bit of bookkeeping and pinging everyone with broken packages. I think the most annoying thing about haskellPackages is that as the one doing the merge you have to arbitrarily judge which packages matter and which don‘t. I want this process to become easier.
Having a joint jitsi call for merging together or at least once in a while seems nice.
One idea for load balancing: What do you think about aiming for a merge about every 2 weeks and cycling through the merge-manager? I think I might commit to doing this every 6 weeks.
One minor point not to forget: After pushing to master we need to update the Nixos version of packages on hackage. Someone needs access to do that.

I'm not sure who to continue to ask about this though.

Hydra role stuff is documented here (a bit hidden tbh), I think the best person to talk to is @grahamc.

I think since cabal2nix belongs to the nix org, we already have commit rights there.

No, the commit access is broken up into several teams and nixpkgs and cabal2nix are separate teams (if at all).

I think the most annoying thing about haskellPackages is that as the one doing the merge you have to arbitrarily judge which packages matter and which don‘t. I want this process to become easier.

I think we need to formalize this process: I think we need to a) compile a list of important™ haskellPackages and b) if possible add basic tests for them (via pkgs/tests). Ideally this would be a single nix file the merge manager would run to have a sanity check on the state of haskell-updates. I have been meaning to work towards such a thing for a while and could commit myself to implementing that.

One idea for load balancing: What do you think about aiming for a merge about every 2 weeks and cycling through the merge-manager? I think I might commit to doing this every 6 weeks.

I think rotating the responsibility is a great idea and I'd be up for that as well.
Concerning the frequency: I think we need to consider if this works well with us tracking the nightly solver (I don't have any opinion on this tbh). If we reduce the frequency of the merge we need to consider merging fixups into master. Maybe we could add an additional configuration.nix file which allows usage of markUnbroken and such, so we don't have to regenerate the package set every time. This file would then be cleaned up again as part of every merge.

I think we need to formalize this process: I think we need to a) compile a list of important™ haskellPackages and b) if possible add basic tests for them (via pkgs/tests). Ideally this would be a single nix file the merge manager would run to have a sanity check on the state of haskell-updates. I have been meaning to work towards such a thing for a while and could commit myself to implementing that.

This sounds great. I am here if you want to bikeshed the details! Maybe we can make a hydra-job out of it?

Concerning the frequency: I think we need to consider if this works well with us tracking the nightly solver (I don't have any opinion on this tbh). If we reduce the frequency of the merge we need to consider merging fixups into master. Maybe we could add an additional configuration.nix file which allows usage of markUnbroken and such, so we don't have to regenerate the package set every time. This file would then be cleaned up again as part of every merge.

Huh, that sounds bad. I think regenerating haskell-packages is not that hard, right? Also, merging fixups into master should be rare, right?

This sounds great. I am here if you want to bikeshed the details! Maybe we can make a hydra-job out of it?

Yeah that should be fairly easy, either a job with multiple constituents (like the channel jobs), or we just use the test to the list of jobs the hydra jobset builds. I'd have to check how the jobset is setup to figure out the best way.

Huh, that sounds bad. I think regenerating haskell-packages is not that hard, right? Also, merging fixups into master should be rare, right?

It isn't, but extra hassle (you need to clone cabal2nix, clone two repositories into specific subdirectories, run the script, …) and currently not very well known or documented. My intention is mostly to make the story of X isn't working in haskellPackages, but it's easy to fix better and ideally remove the n weeks until merge delay. However, this has also the risk of increasing the amount of work for us, so maybe this is something for later.

I can't commit more time at the moment. I'll think about it though.

It isn't, but extra hassle (you need to clone cabal2nix, clone two repositories into specific subdirectories, run the script, …)

That can be solved by pinning the tools used in nixpkgs itself. I've created https://github.com/NixOS/cabal2nix/issues/431 for this, some time ago. It didn't get much support at the time because it didn't match the cabal2nix-repo-based workflow. We should support both though. One because it's easy to use for casual contributors and the other because cabal2nix/hackage2nix need to be developed as well.

:+1: for the initiative.
I created a Haskell team and put @peti as the maintainer so he can invite other people: https://github.com/orgs/NixOS/teams/haskell/members
That team has maintainer access on the cabal2nix project.
/cc @kosmikus and @shlevy who were involved in cabal2nix.

I'd be happy to help at times. But I think I'm already checking some haskell builds on staging-next branches :)

Except for the fixing of breakages - which is the majority of work, we could automate the process with GitHub Actions.
I have admin access to Hydra so I can help with restarts, etc.

If you want I can also enable discussions on the teams page at https://github.com/orgs/NixOS/teams/haskell. If that's helpful to coordinate.

Let me see if I can summarize the discussion up until now with respect to our suggested workflow moving forward.
The members of the Haskell team (for now, me, @sternenseemann, and @maralorn) will take turns fixing up the haskell-updates branch and merging it into master.  This process will take place over the span of two weeks.  The following steps describe this process:

Opening a PR for merging the haskell-updates branch into master.
Immediately after opening the PR, do the following:

Update the stackage resolver in pkgs/development/haskell-modules/configuration-hackage2nix.yaml and the pins for all the packages from Stackage Nightly.  Commit this. (I think we don't have a script to do this automatically, but we probably should.)
Update the all-cabal-hashes file in pkgs/data/misc/hackage/default.nix to point to the latest state of Hackage.  Commit this.  (Is there a script that does this automatically??)
Push these two commits to GitHub on the haskell-updates branch.

hackage2nix will be automatically run by a GitHub action.  The changes here will be automatically be committed and pushed to the `haskell-updates branch by the GitHub action.  (I imagine in the first week or month or so, we may not have this github action, so we may have to do this by hand.)
This should cause the hackage-updates jobset on Hydra to run.  This might take a couple hours to finish?  After this runs, we should get a list of newly broken Haskell packages, along with evaluation errors.
(This might happen a day or so later...) For any packages that are newly broken on Hydra, check if there is a maintainer for the package in pkgs/development/haskell-modules/configuration-hackage2nix.yaml.  If there is, then ping the maintainer on the PR opened in step 1.  (This seems like it would be quite annoying to do in practice, so it would be great to automate this.  I imagine this would be non-trivial to automate though.)
Over the course of the next two weeks, people will send PRs to the haskell-updates branch which unbreak packages.  These may be packages that were newly broken, or they may be packages that have been marked broken for a long time.
After any PRs merged into haskell-updates, hackge2nix will run automatically in the GitHub action.  This is quite convenient.  Alternatively, people sending PRs are also free to run hackage2nix within their PRs.
At the end of the two weeks, the maintainer is responsible for doing the following:

Fix up any evaluation errors on haskell-updates.  See this blog post for a way to check this locally.  We absolutely don't want any evaluation errors on master.
Mark any packages Haskell packages that are still failing to build as broken.  Make sure to commit and push.  The GitHub action will run which will regenerate hackage-packages.nix and mark things as newly broken.  Possibly wait for Hydra to run and confirm that there are now no non-broken packages that fail to build.
Merge the above haskell-updates PR into master.

What do you think about blessing on room on an arbitrary chat-medium where we can do low-level coordination?

This sounds like a good idea!  My only request is that we don't do IRC.
A couple possibilities:

the GitHub team discussions page that zimbatm offered
a channel on the NixOS discord (I think @maralorn is on discord, but I don't know about @sternenseemann)
some slack channel that we can all join?

I don't really have a strong opinion. Although it might be nice not to have something that is completely private, so that other people interested in what we're doing can follow along.

Excellent write-up @cdepillabout. This is very close to what I would envision. Couple of remarks:

Immediately after opening the PR, do the following: ,

Yeah, we should have scripts for that. Working on that right now based on #86699.

hackage2nix will be automatically run by a GitHub action. The changes here will be automatically be committed and pushed to the `haskell-updates branch by the GitHub action. (I imagine in the first week or month or so, we may not have this github action, so we may have to do this by hand.)

I am not one hundred percent sure about the github action anymore. It sounds convenient, but when we can tell people please run "maintainers/scripts/haskell/regenerate-haskell-packages.sh" as part of your commit.  Also we clutter the commit history with less commits. But we can of course add this github action at any point in time.

(This might happen a day or so later...) For any packages that are newly broken on Hydra, check if there is a maintainer for the package in pkgs/development/haskell-modules/configuration-hackage2nix.yaml. If there is, then ping the maintainer on the PR opened in step 1. (This seems like it would be quite annoying to do in practice, so it would be great to automate this. I imagine this would be non-trivial to automate though.)

I am actually trying right now to automate this. It ought to be possible. But no promises.

One question I have is whether or not we should merge master into the hackage-updates branch at any point during this process.

Yes that is a good point. My thoughts are this:

I am not sure about the exact timings anyway, e.g. when to fork off haskell-updates and when to update hackage. If we do both immediately after the merge we guarantee that we are always two weeks out-dated. I could imagine rather fixing the start of the PR than the end. e.g. every two weeks. Fork master, do the update, initiate stabilisation period and merge as soon as everything looks fine.

No matter how we do it, merging master in about two or three days before intended merge is a good idea. (But also not super important, we can never be sure that no one master breaks our packages anyway.)

I only have a strong opinion that merging master into haskell-updates should generally be avoided shortly before merging because it increases the risk of mass hydra rebuild and flying blind.

peti sometimes did rebase stuff to make the history cleaner. I think we should just not do that. If we don‘t have the nightly cron-job our history will look nicer anyways.

This sounds like a good idea! My only request is that we don't do IRC.
A couple possibilities:

the GitHub team discussions page that zimbatm offered
a channel on the NixOS discord (I think @maralorn is on discord, but I don't know about @sternenseemann)
some slack channel that we can all join?

My favorite would be a matrix channel. But anything that we can bridge to matrix would be fine. So discord would be okay for me.

I don't really have a strong opinion. Although it might be nice not to have something that is completely private, so that other people interested in what we're doing can follow along.

100% on-board. Let’s be as transparent as possible. Let’s invite everyone to join! Let’s make good documentation! And let’s distribute the maintenance on as much shoulders as possible! (It‘s not even that I don‘t want to do the work, it’s that together the results will be better!)

Oh, and I think there was a suggestion that we do a video chat or something in the next couple of weeks to make sure we're all on the same page with the above workflow. That definitely sounds like a good idea.
What time zones are you guys in? I'm in JST (UTC+9).

sterni and I are both CEST (UTC+2). I guess the question is: What is the latest that works for you? I guess something like 13:00 UTC on a weekend might work?
(Side note: Sorry for everyone being spammed by this issue.)

haskell-updates on hydra just needs one file from peti/ci. We can just move that file to nixpkgs when we feel like it and then just once someone needs to redirect the hydra job to the new config file.

I'm looking into redoing that and moving it into pkgs/top-level/release-haskell.nix.

One question I have is whether or not we should merge master into the hackage-updates branch at any point during this process. I'm worried that the haskell-updates branch/PR will effectively be behind master by 2 weeks by the point we merge it in, so it may cause mass rebuilds in Haskell packages on master. For example, if one of the dependencies of GHC has changed.

We need to merge master in definitely, as any dependency of GHC changing would cause a mass rebuild. I'd say we add haskell-updates to https://github.com/NixOS/nixpkgs/blob/master/.github/workflows/merge-staging.yml / make a similar action for this.

I only have a strong opinion that merging master into haskell-updates should generally be avoided shortly before merging because it increases the risk of mass hydra rebuild and flying blind.

Note that this mass rebuild is unavoidable. If we push back merging master, the mass rebuild will happen after the merge. I think with the live streams being a thing of the past for now, we don't have to make a specific time window, so we can just wait for hydra to finish and merge whenever everything is green on hydra to our satisfaction.

Update the all-cabal-hashes file in pkgs/data/misc/hackage/default.nix to point to the latest state of Hackage. Commit this. (Is there a script that does this automatically??)

I'm a bit concerned about this to be honest. It does alleviate some annoyances with the update process, but we then need some way to update specific packages to a version newer than the snapshot we chose: For instance due to security updates or if a new release fixes some build issue we encountered.
I'm not sure if merging a two week old hackage snapshot is always a good idea, but I'm happy to try it out at least.

I only have a Discord account of the three, but I guess if we make this an official haskell nixpkgs coordination things something open seems more appropriate.

Note that this mass rebuild is unavoidable. If we push back merging master, the mass rebuild will happen after the merge. I think with the live streams being a thing of the past for now, we don't have to make a specific time window, so we can just wait for hydra to finish and merge whenever everything is green on hydra to our satisfaction.

This is true.

I'm a bit concerned about this to be honest. It does alleviate some annoyances with the update process, but we then need some way to update specific packages to a version newer than the snapshot we chose: For instance due to security updates or if a new release fixes some build issue we encountered.

My suggestion would be, that if that’s needed, we will just bump it once more and see what happens. No reason to be dogmatic about only updating once. I just don‘t want to update automatically everyday so that we constantly get new problems.

We had a meeting on the NixOS Discord today.  Here is a rough list of what we discussed:
Scripts for Maintanance
In https://github.com/NixOS/nixpkgs/pull/121391 @maralorn put together a few scripts for various maintainance tasks related to updating Haskell packages here in Nixpkgs.  For example, bumping the Stackage packages, updating the Hackage pin, updating the Hackage packages.
There was a question as to whether or not we want separate scripts that also make a Git commit after bumping/updating.  I don't know if we came to a very strong consensus, but it did seem like it might be nice to make sure our commit messages are always the same.  Although it also seemed like we might want to run each script without creating a new commit.
One way to solve this would be to add an optional --do-commit flag to each script.  If you pass this flag, then the script does a git commit at the end of the script with a good commit message.
Hydra
@sternenseemann put together a PR that updates the Hydra release file for the haskell-updates branch: https://github.com/NixOS/nixpkgs/pull/121407
We need someone to change the configuration fo the haskell-updates jobset on Hydra.  After merging #121407, @sternenseemann is going to try to get @grahamc to do this.  (@sternenseemann, if you can't get a hold of @grahamc, please raise that issue here and we'll try to figure out how we can make this happen)
It appears that if you sign into Hydra with your GitHub account, you are able to trigger a re-evaluation at any time.  The Haskell maintainers will probably frequently use this feature when maintenance work.
Workflow
We had various discussions on what we want our workflow to look like moving forward.  It seems like we all don't want to have a lot of hard-and-fast rules, but try to play things by ear.  Here is what we did decide to:

The people in the @NixOS/haskell team would each take responsibility of merging haskell-updates into master once every two weeks.  This means that during every two week period, the person who's turn it is needs to merge haskell-updates into master at least once.  It doesn't matter when you do it during the two week period.
Since there are 3 of us in the team, we would each be ultimately responsible for this once every 6 weeks.
If one of us gets busy, then he will let the other team members know and someone else will fill in.
If you update hackage-packages.nix and one of the Haskell packages breaks that has a maintainer, then you have to ping that maintainer on the haskell-updates PR and tell them it is broken.  You need to give them at least 7 days to respond before merging haskell-updates to master.  If they don't respond to fix the problem, then you mark their package as broken.
You don't necessarily have to wait 7 days if you're willing to fix the problem yourself.  Also, you're free to merge haskell-updates into master more than just once every two weeks as long as you're careful to make sure nothing big has started to break.
You can only merge haskell-updates into master when the mergeable job from https://github.com/NixOS/nixpkgs/pull/121407 has passed on Hydra.
At the start of your 2 week period, you should probably bump the Stackage Nightly version along with Hackage packages.

The initial rotation will go:

@maralorn
@cdepillabout
@sternenseemann

Documentation
@maralorn created an issue about documentation: https://github.com/NixOS/nixpkgs/issues/121403.
We also discussed that at the very least we'd like to get our workflow documented for the haskell-updates branch.  @maralorn might work on this during the next two weeks.  If he doesn't get to it, then I will do it during my two weeks.
Future Improvements
We talked about various future improvements we'd like to make.  This is mostly stuff that would help with our maintenance of Haskell stuff in Nixpkgs.

@maralorn is currently writing a script for querying hydra for broken builds.  The script then queries maintainers for those packages. It will produce a message that can be copy-and-pasted to GitHub to warn maintainers that their Haskell packages are now broken.  @maralorn also wants to expand the maintainers in configuration-hackage2nix file.  Moving forward, we'd like to almost completely stop caring about Haskell packages in Nixpkgs without maintainers (well, except for the "important" packages like lens, aeson, etc).

@maralorn is writing a related script for automatically marking Haskell packages as broken based on their most recent hydra status.

We discussed that it would be nice to rewrite hackage2nix to automatically mark packages as broken that have transitive dependencies that are broken.
There are a couple problems with this:

We could just not do this.  This means that if a package is only broken because it has transitive dependencies that are broken, then we don't mark it as broken.  This is apparently okay in terms of Hydra evaluations.  However, it makes Hydra's output harder to understand (since it is hard to tell exactly which packages need to be fixed).
When a package's transitive dependencies are broken, sometimes it is just a dependency of the tests that is broken.  However, we may disable the tests in configuration-common.nix, so hackage2nix doesn't know that the tests can be ignored.  One way to work around this is to have a list of packages that need tests disabled in configuration-hackage2nix.yaml, but we don't want this file to get too complicated.

We think it would be possible to jailbreak packages automatically in hackage2nix.  There are of course dangers with doing this.

We should be able to auto-generate a list of "support levels" for Haskell packages.  For instance, packages with a maintainer that compile successfully, packages without a maintainer but known to compile, packages without a maintainer but jailbroken, packages without a maintainer and with tests checks disabled (this is the scary category).
However, we're not sure who exactly would want this information, so the priority of this is probably lower than the other potential improvements.

Great write up!!
One thing we didn‘t discuss much, but is very important to me: We should try to be very open and transparent about our process. I would also very much like for more people to join the rotation. 3 people is probably fine. more than 5 or 6 is probably too much. But we should communicate a standing invitation for more people to join.
Besides that there are of course a lot of other things people can do to help with haskellPackages. Documentation would probably be one of the biggest pain points.

One thing I've wished for is for the different language ecosystems to converge and for that effort, I wonder if pkgs/development/haskell-modules/configuration-common.nix (which I find too abstract, I have no idea what it does)  could be renamed to pkgs/development/haskell-modules/overrides.nix similar to  pkgs/misc/vim-plugins/overrides.nix / pkgs/development/lua-modules/overrides.nix ?
same for pkgs/development/haskell-modules/hackage-packages.nix , it could become pkgs/development/haskell-modules/generated-packages.nix

@teto That sounds like a good idea!  Please feel free to open a separate issue for this and tag it haskell.

We've basically figured out how we want to move forward with the Haskell infrastructure, so I am going to go ahead and close this issue.
Our workflow is more-or-less described in the following places:

Some formal documentation for our workflow: https://github.com/NixOS/nixpkgs/pull/122728
Some notes from maralorn that the above documentation is based on: https://github.com/NixOS/nixpkgs/pull/122510#issuecomment-839254154

If anyone had any other suggestions about the Haskell infrastructure in Nixpkgs, please feel free to create separate issues for them.


fix(artifacts): Make artifacts cross-platform compatible. Fixes #3682

This partially fixes #3682 problems with GCS on Windows, which were related to filepath handling. I checked all calls of filepath.Clean().
For the S3 problems stated in the #3682 we first need to merge https://github.com/argoproj/pkg/pull/27 and afterwards create a new git tag in argoproj/pkg so that i can create a PR in this repo to update the module for Argo workflows.
Checklist:

[x] Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
[x] The title of the PR is (a) conventional, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
[x] I've signed the CLA.
[x] I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged. - not needed?
[x] My builds are green. Try syncing with master if they are not.
[x] My organization is added to USERS.md. - not needed

ready for review

For testing I created a test function within workflow/executor/executor_test.go and used a real artifact S3/GCS repo. You then need to store the secret files locally and the artifacts get downloaded locally as well. Functions looks similar to this:
func TestCrossPlattformArtifacts(t *testing.T) {
  fakeClientset := fake.NewSimpleClientset()
  mockRuntimeExecutor := mocks.ContainerRuntimeExecutor{}
  templateWithOutParam := wfv1.Template{
    Inputs: wfv1.Inputs{
      Artifacts: []wfv1.Artifact{
        {
          Name:             "samedir",
          Path:             "/argo/inputs/artifacts",
          ArtifactLocation: wfv1.ArtifactLocation{
            /*
            GCS: &wfv1.GCSArtifact{
              GCSBucket: wfv1.GCSBucket{
                Bucket:                  "mlipbucket1",
                ServiceAccountKeySecret: corev1.SecretKeySelector{Key: "serviceAccountKey", LocalObjectReference: corev1.LocalObjectReference{Name: "my-gcs-credentials"}},
              },
              Key:       "artifact-passing-windows-nqfqp/artifact-passing-windows-nqfqp-2841789326/foobar",
            },*/
            S3: &wfv1.S3Artifact{
              S3Bucket: wfv1.S3Bucket{
                Endpoint: "<IP_ADDRESS>:9000",
                Bucket:   "my-bucket",
                Insecure: func() *bool { b := true; return &b }(),
                AccessKeySecret: corev1.SecretKeySelector{Key: "accessKey", LocalObjectReference: corev1.LocalObjectReference{Name: "my-minio-cred"}},
                SecretKeySecret: corev1.SecretKeySelector{Key: "secretKey", LocalObjectReference: corev1.LocalObjectReference{Name: "my-minio-cred"}},
              },
              Key:      "artifact-passing-windows-27z9t/artifact-passing-windows-27z9t-2792197756/foobar",
            },
          },
          //Archive: &wfv1.ArchiveStrategy{None: nil},
        },
      },
    },
    Container: &corev1.Container{},
  }
  we := WorkflowExecutor{
    PodName:            fakePodName,
    Template:           templateWithOutParam,
    ClientSet:          fakeClientset,
    Namespace:          fakeNamespace,
    PodAnnotationsPath: fakeAnnotations,
    ExecutionControl:   nil,
    RuntimeExecutor:    &mockRuntimeExecutor,
    mainContainerID:    fakeContainerID,
  }
  mockRuntimeExecutor.On("GetFileContents", fakeContainerID, "/path").Return("has a newline\n", nil)
  we.LoadArtifacts()
}

Thanks. I'll try my best to support.
Should I update the argoproj/pkg module in this PR as well or create a new one?

If you need the changes I think you need to run go get<EMAIL_ADDRESS>

Yes, I just wanted to know if I should update the go.mod within this PR so that Argo uses the new version with the other fixes for #3682 or if I should create another PR for that.

Please add that change into this PR.

v0.2.0 is was updated in another PR. Mergingi.


Fix set_log_level after logging changes in CASSANDRA-10241

In CASSANDRA-10241, we introduced a separate debug.log. As part of this, a logger for org.apache.cassandra was added in addition to the root level. In order to preserve the existing set_log_level semantics in CCM, we need to also set the log level for this org.apache.cassandra logger.

Thanks!


Make an author dashboard with separate dB for author

Make author dashboard so author can select their books and ratings

Sir, I would like to work on this feature, Please assign this issue to me.
Also can you give me some detailed specifications, Is it just viewing ratings of the his books and adding more books that he wrote in the db.

Author need to know about his book ratings and analysis like top 5 rated books

How many user have reviewed it
Use books ,users table to find analytics like that
Ex : which age user is liking books written by author
Their location

It should give him a complete analysis and you can add the things whatever you wish but it should be a valid one.

What are the best top 5 reviews positive and negative for an author .

So sorry for the delay sir, College is eating up every ounce of time I have but I have started working on it.
Also I just wanted to ask if it was ok if I used canvasjs.com canvasjs's api to plot some relevant graphs.

One more thing, how will the author be able to access this page should I create a signup route for him as well if yes they how will we link and verify them with the books table. I am open to any advice that you can give me.
Thank you sir

@Akshay6890 sir please resolve my query as well

this is an example :
https://user-images.githubusercontent.com/59139142/113479853-f3c61f00-94ae-11eb-97f6-7c6f7f7360a7.mp4
@vikasdo sir and @Akshay6890 sir this what I have done up till now
Now can you tell me 2 things:

How will author be able to access this page (I have just putted in name to search)
what other features should I add

you can implement a feture while login ->login as user ,login as author,  and show only book images with title  when he click on them

how will we verify the login credentials with the authors in books table

@VishvamPorwal  and add metrics like

what are  TOP 5 rated books?
A pie chart of user proportion by age or gender ?

how will we verify the login credentials with the authors in books table
there is no username or password for authors in books table
nor there is any other author table which is linked to it

yes you can create a separate table and add default password for all of them as of now

@VishvamPorwal and add metrics like

what are  TOP 5 rated books?
A pie chart of user proportion by age or gender ?]
and more of metrics on what data available to you and what you can represent

You can see reference of user dashboard of our project

how will we verify the login credentials with the authors in books table
there is no username or password for authors in books table
nor there is any other author table which is linked to it

yes you can create a separate table and add default password for all of them as of now

all of them there are literally 100's of entries in books and even if i were to create another table for authors I will still have to connect every entry of books with a foreign key to the author table

@VishvamPorwal and add metrics like

what are  TOP 5 rated books?
A pie chart of user proportion by age or gender ?]
and more of metrics on what data available to you and what you can represent

we are not taking gender at the time of signup but I can do everything else for sure

@VishvamPorwal and add metrics like

what are  TOP 5 rated books?
A pie chart of user proportion by age or gender ?]
and more of metrics on what data available to you and what you can represent

we are not taking gender at the time of signup but I can do everything else for sure

Gender of user I mean not author

You can take user who has rated the books of author and segregate the user

@VishvamPorwal and add metrics like

what are  TOP 5 rated books?
A pie chart of user proportion by age or gender ?]
and more of metrics on what data available to you and what you can represent

we are not taking gender at the time of signup but I can do everything else for sure

Gender of user I mean not author

Yes I am talking about user as well

do you want me to add gender as well??

do you want me to add gender as well??

In the database I mean at the time of user signup

do you want me to add gender as well??

In the database I mean at the time of user signup

If you want to show metrics on gender  you can add


[red-knot] Move module-resolution logic to its own crate

Summary
This PR moves module-resolution logic to its own crate. It attempts to make the minimal possible changes to do that, so that future PRs can build on top of it. As a result, there's a few things that this PR makes pub that probably shouldn't be pub in the long-term.
The PR... nearly compiles. I may need @MichaReiser's help for the remaining compile errors.

I took a quick look.

You have to remove the resolver "ingredients" from the semantic Jar, they're already part of the resolver Jar.
You have to call db.upcast() when calling resolver queries from the semantic crate

Subject: [PATCH] Don't lazily infer imports because it adds an AST dependency to the module looking up the definition
---
Index: crates/red_knot_python_semantic/src/types/infer.rs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/crates/red_knot_python_semantic/src/types/infer.rs b/crates/red_knot_python_semantic/src/types/infer.rs
--- a/crates/red_knot_python_semantic/src/types/infer.rs	(revision 63049cd8fc5a8f9d23fdac415bfd4b8622fff887)
+++ b/crates/red_knot_python_semantic/src/types/infer.rs	(date<PHONE_NUMBER>963)
@@ -358,7 +358,7 @@
             } = alias;

             let module_name = ModuleName::new(&name.id);
-            let module = module_name.and_then(|name| resolve_module(self.db, name));
+            let module = module_name.and_then(|name| resolve_module(self.db.upcast(), name));
             let module_ty = module
                 .map(|module| self.typing_context().module_ty(module.file()))
                 .unwrap_or(Type::Unknown);
@@ -384,7 +384,8 @@
         let import_id = import.scope_ast_id(self.db, self.file_id, self.file_scope_id);
         let module_name = ModuleName::new(module.as_deref().expect("Support relative imports"));

-        let module = module_name.and_then(|module_name| resolve_module(self.db, module_name));
+        let module =
+            module_name.and_then(|module_name| resolve_module(self.db.upcast(), module_name));
         let module_ty = module
             .map(|module| self.typing_context().module_ty(module.file()))
             .unwrap_or(Type::Unknown);
Index: crates/red_knot_python_semantic/src/db.rs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/crates/red_knot_python_semantic/src/db.rs b/crates/red_knot_python_semantic/src/db.rs
--- a/crates/red_knot_python_semantic/src/db.rs	(revision 63049cd8fc5a8f9d23fdac415bfd4b8622fff887)
+++ b/crates/red_knot_python_semantic/src/db.rs	(date<PHONE_NUMBER>963)
@@ -14,13 +14,9 @@

 #[salsa::jar(db=Db)]
 pub struct Jar(
-    ModuleNameIngredient<'_>,
-    ModuleResolverSearchPaths,
     ScopeId<'_>,
     PublicSymbolId<'_>,
     symbol_table,
-    resolve_module_query,
-    file_to_module,
     scopes_map,
     root_scope,
     semantic_index,


Redesign extension and dataframe registry

Codecov Report
Base: 100.00% // Head: 99.97% // Decreases project coverage by -0.02% :warning:

Coverage data is based on head (f46cde4) compared to base (7d78bf6).
Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head f46cde4 differs from pull request most recent head 2285f94. Consider uploading reports for the commit 2285f94 to get more accurate results

Additional details and impacted files
@@             Coverage Diff             @@
##            master     #375      +/-   ##
===========================================
- Coverage   100.00%   99.97%   -0.03%
===========================================
  Files          110      110
  Lines        10367    10370       +3
===========================================
  Hits         10367    10367
- Misses           0        3       +3

Impacted Files
Coverage Δ

fugue/workflow/workflow.py
100.00% <ø> (ø)

fugue/extensions/creator/__init__.py
100.00% <100.00%> (ø)

fugue/extensions/creator/convert.py
100.00% <100.00%> (ø)

fugue_test/builtin_suite.py
100.00% <100.00%> (ø)

fugue/extensions/_utils.py
96.15% <0.00%> (-3.85%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Ctrl+Z does not work without the Undo button

Chrome 53, Trumbowyg 2.3.0

Configure Trumbowyg without Undo button.
Type some text
Use Ctrl+Z to undo last changes

Expected: Textarea values is updated.
Actual: Textarea still contains value as before Ctrl+Z.
As workaround we added the Undo button and hided it.

Undo feature is not stable and the button only works on Blink/V8 browsers.

Thanks for explanation. Are you planning to improve this logic closer time or it is not a priority feature?

That's not a priority.

Since 2.10.0, you can use history plugin to get an improved and cross-browser history management.
https://alex-d.github.io/Trumbowyg/documentation/plugins/#plugin-history


cursor() on first frame in IntelliJ breaks the sketch

probably a bug - the user should be able to call noCursor at any time...

wrighter is getting this error

i mean cursor(), not noCursor()

can't reproduce, using noCursor() or cursor() in setup() or draw()

I must try it in the pde!

I didn't have it in pde

Could you post some runnable example that would let me see this error?


rm unused models

JIRA ticket number
Context
we have some old unused models which need pruning
Changes proposed in this pull request
rm old completion page models
Guidance to review

Can we remove the associated factories as well? Looks like some may still be kicking around:
https://github.com/DFE-Digital/get-teacher-training-adviser-service/blob/master/spec/factories/uk_new_primary_completion.rb


RequireJS fails ~ missing amd support

Currently we use "what-input" in version "2.1.1". Because it is the last working version for our development process with requirejs.
Version 3 and up loaded uncorrectly. Unfortunately there is no error message ...
Could you try to import dist files via requirejs …

Hi @artursopelnik, I'll take a look into it.

Hey @artursopelnik, I gotta admit, this is tripping me up. In the current version I'm supporting UMD via Webpack. Do you have any suggestions for a UMD/AMD compatible wrapper?

Sorry it was my mistake. You have to be include it with the same module name.
Before:
'what-input': '../../node_modules/what-input/dist/what-input',
After:
'whatInput': '../../node_modules/what-input/dist/what-input',
It is unusual that filename and module name are different. Maybe it makes sense to adjust.

Ah, yes. Good find! I don't want to make a change that would break this for others so I'll add how to do this to the README and keep it in mind for the next major update. Thanks!


HLSL unbounded array resources documentation may be wrong

https://docs.microsoft.com/en-us/windows/win32/direct3d12/resource-binding-in-hlsl
If I declare an unbounded SRV array in the root signature (by setting D3D12_DESCRIPTOR_RANGE1.NumDescriptors to -1) and in HLSL (by having Texture2D<float4> g_pMaterialTextures[] : register(t0);) the debug layer will complain with
D3D12 ERROR: ID3D12CommandList::DrawIndexedInstanced: Static Descriptor SRV resource dimensions (D3D12_SRV_DIMENSION_UNKNOWN) differs from that expected by shader (D3D12_SRV_DIMENSION_TEXTURE2D) at register 128 space 0. ...

However, the documentation clearly states that this is allowed:

An array resource may have an unbounded size, which is declared by specifying the very first dimension to be empty, or 0:
Texture2D<float4> tex1[] : register(t0)
The matching descriptor table could be:
DescriptorTable( CBV(b1), UAV(u0, numDescriptors = 4), SRV(t0, numDescriptors=unbounded)

Obviously, the debug layer cannot validate the descriptors as the actual array size is never explicitly set. The descriptors themselves are a sub range (of size 128 in my case) inside a large descriptor heap so the heap size wouldn't be an indicator either.

If D3D12_DESCRIPTOR_RANGE_FLAG_DESCRIPTORS_VOLATILE is set no validation errors will occur but it's not stated anywhere that this is needed.

@stevewhims can you please take a look at this feedback? Thanks!

Hi, @nico3000, and thanks for filing the issue. I've sent out a request for info about this question to determine whether or not the docs are in need of work.
-Steve

Was the product issue referred to in the final comment ever fixed publicly?  I am encountering what appears to be the same issue with an unbounded SRV array in HLSL and an unbounded descriptor table range in the root signature.


Bug when reading dynamic columns

Hi SHC Team,
I encounter the following bug when read from a table with dynamic tables using the latest commit. Write into table does work as hbase shell show the correct columns:
hbase(main):011:0> scan 'map_bug_test'
ROW                                                      COLUMN+CELL
one                                                     column=cf:a, timestamp=1540453631696, value=?\x8C\xCC\xCD
one                                                     column=cf:b, timestamp=1540453631696, value=@S33
1 row(s)
Took 0.1304 seconds
while reading the same data shows no result:
val readDf = spark.read.options(Map(HBaseTableCatalog.tableCatalog->catalog)).format("org.apache.spark.sql.execution.datasources.hbase").load().show()
+------+----+
|rowkey|data|
+------+----+
+------+----+
readDf: Unit = ()

Full Testcode below:
import org.apache.spark.sql.execution.datasources.hbase._
import org.apache.spark.sql.types._
def catalog = s"""{
|"table":{"namespace":"default", "name":"map_bug_test"},
|"rowkey":"key",
|"columns":{
|"rowkey":{"cf":"rowkey", "col":"key", "type":"string"},
|"data":{"cf":"cf", "col":"", "type":"map<string, float>"}
|}
|}""".stripMargin
val schema = new StructType().add("rowkey", StringType).add("data", MapType(StringType, FloatType))
val dataDS = Seq("""{"rowkey": "one", "data": {"a": 1.1, "b": 3.3}}""").toDS()
val df = spark.read.schema(schema).json(dataDS.rdd)
df.write.options(Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog.newTable -> "5")).format("org.apache.spark.sql.execution.datasources.hbase").save()
val readDf = spark.read.options(Map(HBaseTableCatalog.tableCatalog->catalog)).format("org.apache.spark.sql.execution.datasources.hbase").load().show()

seems like mapType is not supported  but wondering how you are able to write. Check issue #207 "I consider blocking MapType for future use but it shouldn't be a problem because MapType isn't primitive one"

Hi Vivek,
as you see in the linked pull request, this one was merged. We tested the master version and writing works fine, reading doesn't.
Regards,
David

Hi,
I think we found the issue with the code, HBaseRelation.RESTRICTIVE -> HBaseRelation.Restrictive.none was missing in the reading code. Now it seems to work.
One thing which would be great is to get some documentation about these RESTRICTIVE modes, what does family mean, what does none means. But we think we get it now.
Thanks and Regards,
David

@davidvoit  can you share how you set it in the reading code ?

spark.read.options(Map(HBaseTableCatalog.tableCatalog->catalog, HBaseRelation.RESTRICTIVE -> HBaseRelation.Restrictive.none)).format("org.apache.spark.sql.execution.datasources.hbase").load()

Hi @CZuegner ,
Thanks for pointing that.
I try that not working it is not able to recognize RESTRICTIVE from HBaseRelation, it seems the dependency is not the latest of the project.
Is there a process to integrate this new release to a maven project ?

Any comments on @dfossouo 's question? It seems maven repository doesn't have the latest version of the project.

It's been 2 years since @dfossouo comment.
Why the latest build is not being pushed to Hortworks repo?
If it wasn't to be available to the community, can you guys please mention that in README until what it is available and after what it is not?
If this repo is not to be maintained, can original authors mark this repo 'archived' so that people won't spend time considering it for their production level applications?


Zero delta CLS in some Android devices

While making reports on the web vitals data I am gathering I noticed quite a lot of zero delta values for CLS and a pages total CLS was also zero. This concerned me as those entries would pull down the average CLS.
I dug into the data and it only happens on Android 7, 9, 10 or 11 devices using Chrome,  Samsung Internet or Android viewweb. And not all the time.
Could these be true zero CLSs?

Sorry for the slow response (missed the notification for this somehow).
Are you rounding the CLS values before sending them to your analytics system? If so, then I suspect the issue is that the layout shifts are real, but they're so small they get rounded to 0.

I don't do any rounding, but I think Google Analytics does. A bug in GA4 DebugView highlighted that fractional numbers are multiplied by 1000000 and stored as integers.
I did some playing via Data Studio and it looks like values from GA4 are rounded to the nearest 0.000001
Another check of the data and it seems these close to zero CLSs happen on all devices, so that I think was a red herring.


Update packages from repo https://github.com/processone/ejabber {error,bad_eocd}

Hi,
sometime this message comes up on start and i have to restart. After restart its gone.

Same here would like to know what the problem is...

@mwlczk I'm starting Ejabberd using docker from an AWS ec2 instance. What might be the problem, in this case? Thank you.

seems to be an erlang issue:
https://bugs.erlang.org/browse/ERL-196?jql=text ~ "bad_eocd"


Update apache-jena-libs to 4.4.0

Updates org.apache.jena:apache-jena-libs from 4.2.0 to 4.4.0.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.apache.jena", artifactId = "apache-jena-libs" } ]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #16.


Adding exchange to application

Hi I'm having issues when trying to add an exhange.
I followed the step in https://gekko.wizb.it/docs/extending/add_an_exchange.html but after recompiling th e UI the exchange is not added to the dropdownbox. I tried to see what the effect is of the exchange.js files in the exchange folder. When i remove some of the standard exchanges and then recompile the ui the dropdownbox stops working.

Any idea on how to proceed?

Hey @Yamakaza,
Since #750 you don't need to recompile the UI anymore (but instead restart the node process). I will update the documentation to reflect this.
If you added an exchange to the exchange folder that exposes a getCapabilities method, you can start the UI and browse to http://localhost:3000/api/exchanges to get a JSON list of capabilities for every exchange. Can you let me know if that is still working? And if not if there is any error or so?

Hey @askmike thank you for reaching out to me so quick. My exchange has the getCapabilities method but is not showing up in the exchange list. Also I don't seem to find the exchange the api/exchanges.
I tried removing that function from another exchange and restart the node and that removes it from the list. Gekko just reads the JS files in the exchanges folder right? And checks if they have that function?
Can you help me out? :)

Here is method for the api/exchanges call: https://github.com/askmike/gekko/blob/stable/web/routes/exchanges.js
So yes, what it does is: look at all files inside the exchanges folder (that end with .js) and tried to load them. If it is possible to call getCapabilities() on the trader inside it will be added. If you post the exchange file here I can help debug.

Let me know if you have any more information.


Deleted Titles in TMM wouldnt deleted on hdd

Version: 2.9.1
Build: 2016-12-12 17:48
OS: Windows 10 10.0
JDK: 1.8.0_111 amd64 Oracle Corporation
What is the actual behaviour?
The films i marked to delete (in database AND on HDD) wouldnt get deleted after closing and reopening TMM.
On hdd they are renamed as .deletedByTMM*********
What is the expected behaviour?
Normally i expect that the .deletedByTMM*** folders gonna deleted after closing TMM or restarting TMM
Steps to reproduce:
Additional
Have you attached the logfile from the day it happened?
tmm_logs.zip

I was not able to reproduce the creation of these wrong backup folders (.deletedByTMM).
Nevertheless I've included two possible fixes - could you try such a case with the nightly build?
http://nightly.tinymediamanager.org (please do not use the nightly build on productive data)

Hi manuel,
unfortunately i cant test it with the nightly :(
By trying to start the nightly one my system (Explorer.exe crashes -> windows 10 Enterprise) and i dont have another computer to test this :(
Regards
Kathrin

Hi Manuel,
so in managed to make it run and now youre right, it works fine (with the nightly)
The deleted folders (deleted over tmm) were now dropped into a folder deletedbyttm
After closing ttm this folder was deleted automatically
Thanks a lot
Kathrin


add-integrate checkstyle,findbugs and sonar tool to sculture

it will be good enhancement

we used intellij checkstyle.. so it is ok


doc: update instructions for updating DataHub on quickstart

Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[x] Docs related to the changes have been added/updated (if applicable)

@jjoyce0510 removed that warning. Now it is simply adding the instructions for updating locally


Issue luma updater

What model of system are you using?

[ ] Old 3DS
[x] Old 3DS XL
[ ] 2DS (non-XL)
[ ] New 3DS
[ ] New 3DS XL
[ ] New 2DS XL

What is your current Luma3DS version?
If you are running an hourly version of Luma3DS, please manually update to a stable version as I will no longer deal with people who are using hourlies.

[ ] Luma3DS v7.1 + boot9strap v1.0/v1.1
[ ] Luma3DS v8.0 + boot9strap v1.2+
[ ] Luma3DS v8.1 + boot9strap v1.2+
[ ] Luma3DS v8.1.1 + boot9strap v1.2+
[ ] Luma3DS v9.0 + boot9strap v1.2+
[ ] Luma3DS v9.1 + boot9strap v1.2+
[X] Luma3DS v10.0+ + boot9strap v1.2+

What version of Luma Updater are you using?

[ ] v2.0
[ ] v2.1
[ ] v2.1.1
[ ] v2.1.2
[ ] v2.2
[ ] v2.3
[X] v2.4
[ ] v2.5

How are you running Luma Updater?

[X] .cia
[ ] .3dsx + HBL/Rosalina

What problems are you experiencing?
Please tell us what problems you are facing.
I get an option to update the luma updater and when I click update I get update failed and my error message is "Something went wrong while trying to update, see screen below for details." and "Reason for failure: DOWNLOAD FAILED" and thats the only thing I have got.
What is the exact error code you are receiving?
"FATAL: Could not get Location header for 3xx reply"
"Ret code: ded92"
Steps to reproduce

well I just opened it
clicked update
and this

I just read the thing lol


[Feat Req]  Nep bar Toggles

It'd be nice if you could order and have multiple rows of toggles, especially for healing.

Rows have been implemented and are located under nep -> settings


FetchURLfromJQ Renaming Files Incorrectly on Linux

FetchURLfromJQ properly renames the downloaded file in Windows to the UpdateSourceTarget, but treats the new filename as a directory in Linux and fails. Example code below:
{"UpdateStageName": "Base Config Download 2","UpdateSourcePlatform": "All", "UpdateSource": "FetchURL", "UpdateSourceData": "https://github.com/CubeCoders/AMPTemplates/raw/main/ts3servertemplate.ini","UpdateSourceTarget":"teamspeak3-server_win64/ts3server.ini"}

I have observed the same behaviour with FetchURL on Linux

Functioning as intended - it's not meant to change the filename, it just changes what directory it downloads into.


Is there any way to read the cached image file path in react-native-fast-image

@DylanVann I am using react-native fast image to show the images(remote urls) from my server. How can I read the file path of that image.

@DylanVann Is there any way to return the cached image file path in callback function?

same issue

Is there any updates ?

Any update?

#877


Config Groups

Add a feature for the config management module for host groups.
Use Case
be able to assign a host to a group for the purpose of config management. the groups can have configuration assigned to it that is then rendered as part of the hosts configuration.
Details
Groups can be stored in a single model groups with the following fields

id primary key
name name of the group
parent primary id of the parent group or null for group with no parent
config json of groups config
created dtg created
modified dtg modified

Config rendering

[x] function within model to render config

Config rendering can be done via a function that is a recursive lookup. the lookup returns the config of the child group and merges it with the current group overriding any config that conflicts with that of the child group. conflicting settings override precedence is child group then groups from left to right. A hosts config will always override that of any group config.
Config validation

[x] Exception Configuration is invalid

Any configuration will need to have it validated as valid json.
Hosts

[x] new field config_groups

Host group storage will not need a model as the hosts model can be modified to include a single field that is a one to many field. this will allow a host to be assigned to multiple groups.
Requirements

[x] history is maintained
[x] API endpoint for config groups and rendered config

removed the relation with #33

added 5m of time spent

marked this issue as blocking #43

marked this issue as blocking #33

added 23m of time spent

mentioned in commit d422f2feee4a5013ced153c43e0858098890d90b

mentioned in commit fdeae217fa8883031b67df12c1f0f8b06ff92bbd

mentioned in commit 36eb9d34109e7832b72ee0bb5f532a12e1ee7579

marked this issue as blocking #33

mentioned in commit 1e56d8a2abe83526d3d2a80f9b452722c2fffa8e

mentioned in commit 746b7ac747fbb39657912225135dc1d4d4178c8c

marked this issue as blocking #44

mentioned in commit c9982e43be233044937ea673d2afc6d13321803b

mentioned in commit a7d195dfcbd38d14e04b1f45faeba09baca21696

mentioned in commit df27a7dfd365faed6ee1194433e0d8da9499600b

mentioned in commit fff51e38d2503bf4741b3734bfffad6d537fd862

mentioned in commit 7abcc8b6dc86b1e58c5a8bf0e8971043ad8a371e

mentioned in commit 69d07701b2f6a8532a6282a38899776dde77d79e

mentioned in commit c9098f5d2fe1817d7d33b7ffd30aeffa4077cdd2

mentioned in commit 7eee0a26a966809628ddc34a8cdeaae8fe562f54

mentioned in commit 39bfbd25cbc2f936f54600d508cfd8c67a4e023b

marked the checklist item Exception Configuration is invalid as completed

mentioned in commit fc2b47a4cf3b6a0af4ec9b4f8f47e327e64499bd

mentioned in commit 3e2b10a6947f4a11c8e9bf4e9ff4361bf03d08e7

mentioned in commit aeef107026e950b2f27934843c6f4e0f2ad34e8b

mentioned in commit 62e605d4172c114d9d14a6aebf2bc122cee21866

added 13h 35m of time spent

mentioned in commit 5cb155e01f72680c8249690be885f386679d458a

mentioned in commit 9e4ee4e9a666fc6cdaf05bd66dc36000dc7eac98

marked the checklist item new field config_groups as completed

mentioned in commit 55f0db2217e247d7f76edd1c2e81bfd9b7570698

subtracted 13h 35m of time spent

mentioned in commit 1f35f44f20d3f00adaaa371af45be00516451a6c

mentioned in commit 0a17329a710e7f94ea3054857975467236130d1c

marked the checklist item history is maintained as completed

mentioned in commit 23a06be3eb00b34c0b1a60ea769a5783b1f9da21

mentioned in commit<PHONE_NUMBER>e906ae1fd3419a018441de78c5741d

marked the checklist item API endpoint for config groups and rendered config as completed

marked the checklist item function within model to render config as completed