add kubectx and kubens as artifacts to tagged releases

This PR adds a deploy stage to .travis.yml which adds kubectx and kubens to a tagged release

I signed it!
On Sun, Dec 23, 2018 at 12:50 PM googlebot<EMAIL_ADDRESS>wrote:

Thanks for your pull request. It looks like this may be your first
contribution to a Google open source project (if not, look below for help).
Before we can look at your pull request, you'll need to sign a Contributor
License Agreement (CLA).
📝 Please visit https://cla.developers.google.com/
https://cla.developers.google.com/ to sign.
Once you've signed (or fixed any issues), please reply here (e.g. I
signed it!) and we'll verify it.
What to do if you already signed the CLA Individual signers

It's possible we don't have your GitHub username or you're using a
different email address on your commit. Check your existing CLA data
https://cla.developers.google.com/clas and verify that your email is
set on your git commits
https://help.github.com/articles/setting-your-email-in-git/.

Corporate signers

Your company has a Point of Contact who decides which employees are
authorized to participate. Ask your POC to be added to the group of
authorized contributors. If you don't know who your Point of Contact is,
direct the Google project maintainer to go/cla#troubleshoot (Public
version https://opensource.google.com/docs/cla/#troubleshoot).
The email used to register you as an authorized contributor must be
the email used for the Git commit. Check your existing CLA data
https://cla.developers.google.com/clas and verify that your email is
set on your git commits
https://help.github.com/articles/setting-your-email-in-git/.
The email used to register you as an authorized contributor must
also be attached to your GitHub account
https://github.com/settings/emails.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ahmetb/kubectx/pull/110#issuecomment-449628320, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AbPTmY58oVSjohbPzbDWs09SQ1t9uZhCks5u71_9gaJpZM4Zfyz6
.

Why do you need this while you can download the scripts directly from the tree? :)

Force of habbit, I guess. Thought it would be a nice touch...
You can close if you wish :)
Great tool, btw! Thanks.

I'll close. I don't think we need this right now. If we move to language like Go in the future, we'll revisit.


Sort not working with float numbers

When one of the columns has float numbers and we sort by that column the result looks incorrect.
See this example, try to sort by the "Profile Progress" column:
https://codesandbox.io/s/wrong-sort-2n8fk

The default sorter is an alphanumeric one here: https://github.com/tannerlinsley/react-table/blob/127a7fca8754f6289b6bc019dbd1fc4bcdd13187/src/sortTypes.js#L6
You need to use the sortType: 'basic' for now if you want exact sorting for floats: https://codesandbox.io/s/wrong-sort-w60ff

@tannerlinsley unfortunately this option doesn't solve my problem. It works if we have only positive float numbers. On my case I have a mix of positive, negative and null values, the basic doesn't work.
Solved using sortType with a custom sort function.
One note regarding the custom sort function, the documentation  says that there is a desc param:
sortType: String | Function(rowA: <Row>, rowB: <Row>, columnId: String, desc: Bool)

But this value is always undefined.

Maybe it would be better to have the alphanumeric type be named string instead?
I saw the current name and just assumed it handed floats, since floats are "numeric". Whereas if I saw that the default was string that might have helped me track down this bug way earlier.

Maybe it would be better to have the alphanumeric type be named string instead?
I saw the current name and just assumed it handed floats, since floats are "numeric". Whereas if I saw that the default was string that might have helped me track down this bug way earlier.

So I have a unfortunate problem. I have floats that I want to sort, and I want them to be sorted descending by default. The problem is that when I don't use sortType: 'basic', the floats aren't sorted correctly. If I however use sortType: 'basic', then the floats are sorted correctly but they are now no longer sorted by default... How do I solve this?

// Added logic to fix sorting issues with decimal numbers, as DataTable treats all data as strings
{
targets: [1, 2, 3, 4],
render: function (data, type, row, meta) {
if (type == "display") {
return data;
}
return parseFloat(data);
},
},


[Bug]-[5500]:All config based action forms blink when navigating to it

Is there an existing issue for this?

[X] I have searched the existing issues

Description
https://user-images.githubusercontent.com/12022471/193193286-47a03897-ee53-400e-9b47-38fa8ab20d5e.mov
Steps To Reproduce

Create an action with config/saas like S3
Navigate back and forth to it

Public Sample App
No response
Version
Cloud

this applies for all config-based query forms, so 6m usage: 5500 with 100% reach
Stats

Stat
Values

Reach
5500

Effort (months)
0.5

performance seems better so the blink is shorter but it is still there:
https://github.com/appsmithorg/appsmith/assets/94514895/44316070-64aa-4be9-8f12-263004b45285


feat: migration pre-upgrade script

I added the migration but not the conversion script to aries-askar itself.
It is added as a feature (default right now, but can be removed). After a while we can probably remove it from the default features as most people have migrated, but then it can always be opted-in of course.
I was working on the migration script for React Native Sqlite wallets and I was running into a lof of issues regarding sqlite and cryptography dependencies. Every sqlite dependency came with a list of issues and it since we can not patch them (as the sqlite dependency has to be a direct dependency of the wallet), it seemed unfeasible.
I tried to depend on the values / structs / functions from aries-askar itself as much as possible, but I might have created some duplicate code. @andrewwhitehead you can probably spot this a lot better than me here.
The python script could also depend on this function, but it is not required.
The script also only supports indy-sdk sqlite -> aries-askar sqlite. There is no additional support for multiple wallets (although the script can be be called multiple times for this to work, however untested). Postgres has been omitted for now as it did not fit the direct use case. I don't think that adding support for Postgres would be all too complex.
Curious to hear if there are any objections to this PR or if there is a very specific reason why it was not done for the Python script.
cc: @TimoGlastra @andrewwhitehead @swcurran
NOTE: It does not really have a test, yet. I would have to create an indy-sdk wallet and I am not too sure what the quickest way is to do that.

How does this work compare/relate to the work being done in the hyperledger/aries-acapy-wallet-upgrade project?

How does this work compare/relate to the work being done in the hyperledger/aries-acapy-wallet-upgrade project?

There is some overlap between the two. For the migration we have to do a generic (non-(aca-py|afj|af-go)-specific) conversion. This mainly converting the data to the new tables, but not the record transformation.
The record transformation is dependent upon the framework it will be used in, so that part is not implemented here and still has to be done in the framework itself.
The script from aca-py also handles postgres wallets and multi-tenancy properly, this one does not. However, when those fixes are applied in the future, the aca-py script can depend on this function for the preprocessing.
Hope that helps!

@andrewwhitehead There might be some duplicate code (related to kdf and more), but I wanted to modify as little of the rest of the library as possible. This means that removing the script would revert perfectly back to the state before the script without having to remove some code in specific places.
If there is some functionality I can use from askar, without changing anything, let me know.

Running into some issues with the tags. Shouldn't merge this yet.

@andrewwhitehead I merged your PR. Just one question about the name removal of _sqlite as a suffix. Was there a specific reason for this?

@blu3beri I removed the _sqlite suffix because it looks like we (with this change) wouldn't need to change the foreign function interface when Postgres migration support is added.

#89 may need to be merged first to fix the test failure.

I'm not sure why the sqlite rekey test seems to be failing sometimes on Linux. It's running into a database locked error when provisioning a new database with a random filename. It's not related to these changes.


Support remote entities

Looks like all remote entities are always reported as unknown.

Could you share a snippet of a YAML file with the remote entity declaration so we can troubleshoot this issue? Thanks!

Android TV Card:
type: grid
cards:
  - type: horizontal-stack
    cards:
      - type: tile
        entity: media_player.tv_da_sala_samsung
        show_entity_picture: false
        vertical: false
        hide_state: false
        state_content:
          - state
          - last-changed
  - type: custom:android-tv-card
    remote_id: remote.tv_da_sala_samsung
    media_player_id: media_player.tv_da_sala_samsung
    autofill_entity_id: true
    custom_actions:
      up:
        tap_action:
          action: key
          key: KEY_UP
      down:
        tap_action:
          action: key
          key: KEY_DOWN
      left:
        tap_action:
          action: key
          key: KEY_LEFT
      right:
        tap_action:
          action: key
          key: KEY_RIGHT
      center:
        tap_action:
          action: key
          key: KEY_ENTER
        double_tap_action:
          action: key
          key: KEY_RETURN
      power:
        icon: mdi:power
        tap_action:
          action: call-service
          service: media_player.toggle
      home:
        tap_action:
          action: key
          key: KEY_HOME
      back:
        tap_action:
          action: key
          key: KEY_RETURN
      volume_mute:
        tap_action:
          action: key
          key: KEY_MUTE
      input:
        icon: mdi:hdmi-port
        tap_action:
          action: key
          key: KEY_SOURCE
    rows:
      - - back
        - power
        - input
        - home
      - - volume_mute
        - slider
      - - touchpad
title: TV da sala (Samsung)

But this can also be present in automations and everywhere else AFAIK.

I believe remote entities are a relatively new addition to the supported platforms.
It might be a good addition to have a GH workflow verify the platforms that Watchman checks against the complete supported list as part of a test case.
This could help keep track of any new additions, as well as provide opportunity to prune deprecated entity platforms such as proximity.

Will be fixed in 0.6.4


AnimationTree Sub-StateMachines reset to start node

Godot version:

3.1
OS/device including version:

Windows
Issue description:

I don't think my problem is a bug, it's just an issue that I find no way to solve and where I am not able to get help with.
Im trying to animate my player with an AnimationTree:
The state machine of the animation player looks like this:
 and each of the sub-state machines look like this:

and this is the script on the player which controls the state machines:
extends KinematicBody2D

export (int) var walk_speed = 200
export (int) var run_speed = 300
export (float) var secs_till_idle = 5.0

onready var Idle_Timer = get_node("Idle_Checker")

var state_machines = {}
var curr_speed = 0
var move_state = "stop"
var direction = "down"
var velocity = Vector2()

func _ready():
	state_machines["main"] = get_node("AnimationTree").get("parameters/playback")
	state_machines["idle"] = get_node("AnimationTree").get("parameters/idle/playback")
	state_machines["stop"] = get_node("AnimationTree").get("parameters/stop/playback")
	state_machines["walk"] = get_node("AnimationTree").get("parameters/walk/playback")
	state_machines["run"] = get_node("AnimationTree").get("parameters/run/playback")

	Idle_Timer.set_wait_time(secs_till_idle)

func _physics_process(delta):
	_check_movement()
	_check_idle()
	move_and_slide(velocity)

func _check_movement():

	velocity = Vector2()
	if Input.is_action_pressed("move_down"):
		set_direction("down")
		velocity.y += 1

	if Input.is_action_pressed("move_up"):
		set_direction("up")
		velocity.y -= 1

	if Input.is_action_pressed("move_right"):
		set_direction("right")
		velocity.x += 1

	if Input.is_action_pressed("move_left"):
		set_direction("left")
		velocity.x -= 1

	if velocity.length() > 0:
		if Input.is_action_pressed("move_run"):
			set_move_state("run")
		else:
			set_move_state("walk")

	elif move_state != "idle":
		set_move_state("stop")

	if state_machines["main"].is_playing():
		state_machines["main"].travel(move_state)

	_change_all_sub_state_machines(direction) # Doesn't fix anything
	#if state_machines[move_state].is_playing():
	#	state_machines[move_state].travel(move_state + "_" + direction)

	velocity = velocity.normalized() * curr_speed

func _change_all_sub_state_machines(direction):
	if state_machines["idle"].is_playing():
		state_machines["idle"].travel("idle_" + direction)

	if state_machines["stop"].is_playing():
		state_machines["stop"].travel("stop_" + direction)

	if state_machines["walk"].is_playing():
		state_machines["walk"].travel("walk_" + direction)

	if state_machines["run"].is_playing():
		state_machines["run"].travel("run_" + direction)

func _check_idle():
	if move_state == "stop":
		if Idle_Timer.is_stopped():
			Idle_Timer.start()
	else:
		Idle_Timer.stop()

func _on_Idle_Checker_timeout():
	set_move_state("idle")

func set_move_state(new_move_state):
	match new_move_state:
		"stop": set_speed(0)
		"idle": set_speed(0)
		"walk": set_speed(walk_speed)
		"run": set_speed(run_speed)

	move_state = new_move_state

func set_speed(new_speed):
	curr_speed = new_speed

func set_direction(new_direction):
	direction = new_direction

Now the problem is, that if I change from state "stop" in the main state machine to state "walk" for example and from the sub-state machine "stop_left" to sub-state machine "walk_left", the animation with the start marker shows off for one frame in the sub_state machine im traveling in ("walk_down" in this example).
I tried to fix this by traveling every sub-state machine to "[state]_[direction]", but that doesnt fix anything, the animation still gets reset. I have made a video showing it:
https://www.youtube.com/watch?v=i-eziuBjpOU&feature=youtu.be
(This is me mashing the arrow keys in one direction at a time triggering the start node)
So I am questioning the usage of sub-state machines right now. I don't how you would use them without encountering this problem... Maybe I am missing something.
Minimal reproduction project:
Sub-state machine debug.zip
This is the actual project im working on, but its not very advanced so you should see the issue right away.

Here is your project with an update that makes the state machine dance in the right way.
Sub-state machine debug revised.zip
I disabled the autoplay on all the states and added else statements to start the states if they are not running.
	if state_machines["main"].is_playing():
		state_machines["main"].travel(move_state)
	else:
		state_machines["main"].start(move_state)

	_change_all_sub_state_machines(direction) # Doesn't fix anything
	#if state_machines[move_state].is_playing():
	#	state_machines[move_state].travel(move_state + "_" + direction)

	velocity = velocity.normalized() * curr_speed

func _change_all_sub_state_machines(direction):
	if state_machines["idle"].is_playing():
		state_machines["idle"].travel("idle_" + direction)
	else:
		state_machines["idle"].start("idle_" + direction)

	if state_machines["stop"].is_playing():
		state_machines["stop"].travel("stop_" + direction)
	else:
		state_machines["stop"].start("stop_" + direction)

	if state_machines["walk"].is_playing():
		state_machines["walk"].travel("walk_" + direction)
	else:
		state_machines["walk"].start("walk_" + direction)

	if state_machines["run"].is_playing():
		state_machines["run"].travel("run_" + direction)
	else:
		state_machines["run"].start("run_" + direction)

Hmm okay.. thanks for taking a look. Seems a little inconvinient, but it closes the issue.


Move random-related defines to "crypto/rand.h"

commit 7ec5c3109d5cea3d2b264ec1db61fd42ac97ec13
Author: Dr. Matthias St. Pierre<EMAIL_ADDRESS>Date:   Mon Jan 6 01:50:43 2020 +0100
Move random-related defines to "crypto/rand.h"

This fixes commit 01036e2afbe116d608be048ed15930fc885ab2a8, which moved the
DEVRANDOM and DEVRANDOM_EGD defines into rand_unix.c. That change introduced
the regression that the compiler complains about missing declarations in
crypto/info.c when OpenSSL is configured using `--with-rand-seed=devrandom`
(resp. `--with-rand-seed=egd`)

Fixes #10759

commit ee502fd524d52602a61b67425f5e7f1de6f98a11 (HEAD -> pr-move-random-defines-again, msp/pr-move-random-defines-again)
Author: Dr. Matthias St. Pierre<EMAIL_ADDRESS>Date:   Mon Jan 6 02:38:14 2020 +0100
Modify the add_seeds_stringlist() macro to fix a preprocessor error

When OpenSSL is configured using `--with-rand-seed=devrandom`, the preprocessor
reports the following error

    crypto/info.c:104:66: error:
            macro "add_seeds_stringlist" passed 3 arguments, but takes just 2
            add_seeds_stringlist("random-device", { DEVRANDOM, NULL });

The reason why the preprocessor complains about three arguments being passed
is that according to [1], balanced braces in macro arguments don't prevent the
comma from acting as an argument separator:

    3.3 Macro Arguments
    ...
    Parentheses within each argument must balance;
    a comma within such parentheses does not end the argument.
    However, there is no requirement for square brackets or braces to balance,
    and they do not prevent a comma from separating arguments.

Also introduced an iteration pointer `p`, because `dev` is not an lvalue:

    crypto/info.c:78:41: error:
            lvalue required as increment operand
            for (; *dev != NULL; dev++) {

[1] https://gcc.gnu.org/onlinedocs/cpp/Macro-Arguments.html

Side note: the fix was compiled and tested with --with-rand-seed=devrandom. Here's the output:
$ util/shlib_wrap.sh  apps/openssl info -seeds
random-device(/dev/urandom /dev/random /dev/hwrng /dev/srandom)

Merged to master, thanks!
9484b67dfb Modify the add_seeds_stringlist() macro to fix a preprocessor error
2e912f63a4 Move random-related defines to "crypto/rand.h"


useStateSnippet no longer camelCases the callback function. v4.0.3

First of all, love the addition of snippet to input! The latest release is no longer setting the callback to camel case.

Love the extension, just realised how much time it saves me.

released in v4.1.0

@dsznajder snippets dont work in v4.1.0


fix fix the process when erasing elements from std::set

Proposed changes
container::erase(iterator) will return the iterator of the next element of the erased element, ,so we should not call it++ if we erased an element.
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.75 seconds
stream load tsv:          502 seconds loaded<PHONE_NUMBER>9 Bytes, about 142 MB/s
stream load json:         19 seconds loaded<PHONE_NUMBER> Bytes, about 118 MB/s
stream load orc:          64 seconds loaded<PHONE_NUMBER> Bytes, about 16 MB/s
stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select:          29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size:<PHONE_NUMBER>9 Bytes

run p0


Cannot use @fastify/one-line-logger on vercel

Prerequisites

[X] I have written a descriptive issue title
[X] I have searched existing issues to ensure the bug has not already been reported

Fastify version
4.26.2
Plugin version
1.3.0
Node.js version
20.x
Operating system
Linux
Operating system version (i.e. 20.04, 11.3, 10)
Vercel Serverless
Description
I use @fastify/one-line-logger format logs, when deploying fastify as Vercel's serverless function, get below error:
Error: unable to determine transport target for "@fastify/one-line-logger"
at fixTarget (/var/task/node_modules/.pnpm/pino@8.20.0/node_modules/pino/lib/transport.js:144:13)
at transport (/var/task/node_modules/.pnpm/pino@8.20.0/node_modules/pino/lib/transport.js:114:22)
at normalizeArgs (/var/task/node_modules/.pnpm/pino@8.20.0/node_modules/pino/lib/tools.js:316:16)
at pino (/var/task/node_modules/.pnpm/pino@8.20.0/node_modules/pino/pino.js:90:28)
at createPinoLogger (/var/task/node_modules/.pnpm/fastify@4.26.2/node_modules/fastify/lib/logger.js:42:14)
at createLogger (/var/task/node_modules/.pnpm/fastify@4.26.2/node_modules/fastify/lib/logger.js:101:18)
at fastify (/var/task/node_modules/.pnpm/fastify@4.26.2/node_modules/fastify/fastify.js:135:33)
at file:///var/task/server.js:13:13
at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
at async ModuleLoader.import (node:internal/modules/esm/loader:323:24)
Node.js process exited with exit status: 1. The logs above can help with debugging the issue.
INIT_REPORT Init Duration: 602.99 ms	Phase: invoke	Status: error	Error Type: Runtime.ExitError
Unknown application error occurred

Steps to Reproduce

Create a simple fastify api example
Change the logger to @fastify/one-line-logger

Expected Behavior
The fastify api will deploy successfully on vercel

I don't think the pino transport system would work on vercel. You should use it as a stream: https://github.com/pinojs/pino-pretty?tab=readme-ov-file#usage-as-a-stream.

Use pino-pretty as a stream worked, here is the code:
import Fastify from 'fastify';
import pino from 'pino';
import pretty from 'pino-pretty';
const stream = pretty({
  translateTime: 'SYS:HH:MM:ss Z',
  messageFormat: '{msg} {req.method} {req.url}',
  include: 'time,pid,level',
  hideObject: true,
  colorize: false,
});
const logger = pino({ level: 'info' }, stream);

const app = Fastify({ logger });

Thnkas for your reply


Disconnect the Session?

Is the something like a session.Shutdown()? I know there isn't that exact one but a similar functionally (besides killing the loop). I did not see anything in RequestMessages either for something like this. I have the startup code from the Demo shoved into a function that gets called from the HandleCommands loop, and would like to add a disconnect command so a user could log out without closing the window and change the passed arguments to the StartUp. BTW great work on the API, once I get more familiar with it and setup some basic stuff I am hoping to wrap it all up inside an Xna project.

Hit Play and Just Jam

Currently there isn't a "Shutdown" method implemented, although it would be easy to add it.
You can always open a pull request :tongue:

Lol I would love to do it, though I am not sure how you would do it... I think you would need to just disco from the google services....

@Eliminster0
Basically you just need to stop the heartbeat, there isn't any "real" way to logout. I will push an update once I fixed the inventory for deleted stuff.

Yes that what I have did originally, however, the TCP connection with the server persists.
 public static void ShowActiveTcpConnections()         {             Console.WriteLine("Active TCP Connections Endpoints");             foreach(var c in IPGlobalProperties.GetIPGlobalProperties().GetActiveTcpConnections())             {                 if (c.State.Equals(TcpState.Established))                 {                     switch (c.RemoteEndPoint.Address.ToString())                     {                         case "<IP_ADDRESS>":                             Console.WriteLine(Dns.GetHostEntry(c.RemoteEndPoint.Address));                             Console.WriteLine(c.RemoteEndPoint.Address.ToString());                             break;                         case "::1":                         case "<IP_ADDRESS>":                             break;                         default:                             Console.WriteLine(c.RemoteEndPoint.Address.ToString());                             break;                     }                 }             }         }
Now I have looked for a way to revoke token using
` public static async void GiveBackToken(AccessToken token)
{
try
{
var values = new Dictionary<string, string> { { "token", token.Token } };
//var content = new FormUrlEncodedContent(values);
            HttpClient client = new HttpClient();
            var response =
                await client.GetAsync("https://accounts.google.com/o/oauth2/revoke?token=" + WebUtility.UrlEncode(token.Token));
            // var response =
                     await client.PostAsync("https://accounts.google.com/o/oauth2/revoke?token=", content);

            var responseString = await response.Content.ReadAsStringAsync();
            Log.Info(responseString);
        }
        catch (IOException e)
        {
            Console.ForegroundColor = ConsoleColor.Red;
            Log.Fatal(e.StackTrace);
            Console.ForegroundColor = ConsoleColor.Gray;
        }`

But the response comes back "error:" "invalid token".  I am not sure how to get the actual status code from the response message.

Ok, so I "invented" a more hacky way of killing the session without having to stop the heartbeat explicitly  and without handing the token back to the authentication server.
As far as my thoughts on the token turn part is that "https://accounts.google.com/o/oauth2/revoke?token=" is not the right turn in point for the tokens, however this is just speculation.
To start off we have the Disconnect Method:
public static void Disconnect() {
     //If we are connected to the server
     if (IsConnected()) {
        //Start up a new process
        System.Diagnostics.Process.Start("NAME OF YOUR APP.exe", null);
        //and close the current process
        Environment.Exit(1);
    //If we are !connected
     } else {
         //Tell the user to go %$# themselves
         Console.ForegroundColor = ConsoleColor.Red;
         Log.Error("You are not currently logged on.");
         Console.ForegroundColor = ConsoleColor.Gray;
     }
}

and here is the is the IsConnected Method:
public static bool IsConnected() {
    //for all of the current TCP connections
    foreach(var c in IPGlobalProperties.GetIPGlobalProperties().GetActiveTcpConnections()) {
        //if the connection is established and its connect to the server
        if (c.State.Equals(TcpState.Established) &&
            c.RemoteEndPoint.Address.ToString() == "<IP_ADDRESS>")
            //then tell everyone who wished to know about it that we are
            return true;
    }
    //of all the connections none of them are connected to the server
    return false;
}

So there is a very important thing to realize that because this is so hacky you wont be able to run multiple instances of the program that are connected to the server(or at least that what I think because of how the logic is set up). Because my implementation is geared more towards playing the game from console and not so much automation it works, but I would like to leave it the issue open in case a more stable method is created.

A Disconnect() method is implemented in Session.


sleuthkit: 4.6.2 -> 4.6.3

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/sleuthkit/versions.
meta.description for sleuthkit is: '"A forensic/data recovery tool"'.
Release on GitHub
Compare changes on GitHub

Checks done (click to expand)

built on NixOS
Warning: no invocation of /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/sorter had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/mactime had a zero exit code or showed the expected version
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/img_cat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/img_stat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/mmls passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/mmstat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/mmcat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/blkcalc passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/blkcat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/blkls passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/blkstat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/ffind passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/fls passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/fcat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/fsstat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/icat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/ifind passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/ils passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/istat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/jcat passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/jls passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/usnjls passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/hfind passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/srch_strings passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/sigfind passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/tsk_recover passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/tsk_loaddb passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/tsk_comparedir passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/tsk_gettimes passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/fiwalk passed the binary check.
/nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin/jpeg_extract passed the binary check.
29 of 31 passed binary check by having a zero exit code.
26 of 31 passed binary check by having the new version present in output.
found 4.6.3 with grep in /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3
directory tree listing: https://gist.github.com/0a565e706ef01e2f75c2dcebf8c6ad21
du listing: https://gist.github.com/11779084c00a5ee8b2030fc196a2ddb1

Outpath report (click to expand)

Outpath difference report
6 total rebuild paths
2 package rebuilds
2 x86_64-linux rebuilds
2 i686-linux rebuidls
0 x86_64-darwin rebuilds
2 aarch64-linux rebuilds
First ten rebuilds by attrpath
osquery
sleuthkit

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
Or, build yourself:
nix-build -A sleuthkit https://github.com/r-ryantm/nixpkgs/archive/112bbf3b7de7b9b482277c24ad63b507c57d386b.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3
ls -la /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin

cc @7c6f434c for testing.

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: sleuthkit
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: sleuthkit
Partial log (click to expand)
shrinking /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/lib/libtsk.so.13.4.4
gzipping man pages under /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/share/man/
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/lib  /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3/bin
patching script interpreter paths in /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3
basename: invalid option -- 'w'
Try 'basename --help' for more information.
basename: invalid option -- 'w'
Try 'basename --help' for more information.
checking for references to /build in /nix/store/wdfi0lf30m168rqy5d3hqz28hvrphjiy-sleuthkit-4.6.3...

Success on aarch64-linux (full log)
Attempted: sleuthkit
Partial log (click to expand)
gzipping man pages under /nix/store/9kpbmhxgakhxafsxrgqr9phkvp63fi2q-sleuthkit-4.6.3/share/man/
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/9kpbmhxgakhxafsxrgqr9phkvp63fi2q-sleuthkit-4.6.3/lib  /nix/store/9kpbmhxgakhxafsxrgqr9phkvp63fi2q-sleuthkit-4.6.3/bin
patching script interpreter paths in /nix/store/9kpbmhxgakhxafsxrgqr9phkvp63fi2q-sleuthkit-4.6.3
basename: invalid option -- 'w'
Try 'basename --help' for more information.
basename: invalid option -- 'w'
Try 'basename --help' for more information.
checking for references to /build in /nix/store/9kpbmhxgakhxafsxrgqr9phkvp63fi2q-sleuthkit-4.6.3...
/nix/store/9kpbmhxgakhxafsxrgqr9phkvp63fi2q-sleuthkit-4.6.3


action description has odd casing, and perhaps could be more expository

maybe don't capitalize the P in Push? and, as with #12, we have the space to be a bit more expository, if helpful. e.g.
Send a push notification to the specified device, using the IBM pushnotification service.

Changed the action description to Send push notification to all application users or to a specific set of devices


Online CoAP decoder

Similar to cbor.me, an online encoder/decoder tool should be created. Tool should be published to GitHub Pages on commits merged to main.

Changed scope to only include decoding of messages (we don't have much need for an online tool for encoding at this time).
Closing as decoding was implemented via #42.

Changed scope to only include decoding of messages (we don't have much need for an online tool for encoding at this time).
Closing as decoding was implemented via #42.


App Won't Open OSX 10.11.6

Just upgraded to a new iMac. Was previously working perfect on MB Pro.
Tried deleting and re-installing app with no luck.

Solved this by downloading from the App Store instead of from the website...


chat_with_retrieval occur errors when use_flare = True or use_moderation =True

In chat_with_retrieval/app.py, an error occurs if use_flare = True and ,  error if use_moderation =True.
chat_with_documents.py:
FlareChain and ConversationalRetrievalChain have different input and output names.
For FlareChain, user_input and response are used.
if use_flare:
    params = {
        "user_input": user_query,
    }
else:
    params = {
        "question": user_query,
        "chat_history": MEMORY.chat_memory.messages,
    }
response = CONV_CHAIN.run(params, callbacks=[stream_handler])

utils.py:
In def init_memory(), output_key=response should be output_key=response for FlareChain.
return ConversationBufferMemory(
        memory_key='chat_history',
        return_messages=True,
        output_key='answer'
    )

Hey @mrchaos! Thanks for finding that, that's cool! Do you feel like creating another PR?

Hi,
I've been fighting with this issue for a while (and several more that appeared to me as I fixed the initial ones) and finally managed to code a fix. It is implemented with a later version of Langchain (I have upgraded it during the course to avoid issues with OpenAI's deprecated models). I also implemented some changes to avoid deprecation Warnings. I leave my changes below in case it can help anyone.
The only issue I faced is with moderation chain, which seems to have been deprecated by OpenAI and there is no straightforward way to upgrade it on a Windows environment. The rest should run just fine. In terms of performance, the FLARE chain works way worse for me as it tends to hallucinate more and be more "confused" about the documents in its context.
I can do a PR if you like, but I have not tested this with earlier versions of Langchain

langchain==0.1.10
langchain-community==0.0.25
langchain-core==0.1.28
langchain-decorators==0.5.4
langchain-experimental==0.0.53
langchain-openai==0.0.8
langchain-text-splitters==0.0.1

Main changes:
In utils.py:

Upgrade libraries import to avoid deprecation warnings
Add a .lower() for the extension of documents, so we do not get errors with files named in caps (e.g. "FILE.PDF")

"""Utility functions and constants.

I am having some problems caching the memory and the retrieval. When
I decorate for caching, I get streamlit init errors.
"""
import logging
import pathlib
from typing import Any

from langchain.memory import ConversationBufferMemory
from langchain.schema import Document
from langchain_community.document_loaders import (
    PyPDFLoader,
    TextLoader,
    UnstructuredEPubLoader,
    UnstructuredWordDocumentLoader,
)

def init_memory():
    """Initialize the memory for contextual conversation.

    We are caching this, so it won't be deleted
     every time, we restart the server.
     """
    return ConversationBufferMemory(
        memory_key='chat_history',
        return_messages=True,
        output_key='answer'
    )

MEMORY = init_memory()

class EpubReader(UnstructuredEPubLoader):
    def __init__(self, file_path: str | list[str], **unstructured_kwargs: Any):
        super().__init__(file_path, **unstructured_kwargs, mode="elements", strategy="fast")

class DocumentLoaderException(Exception):
    pass

class DocumentLoader(object):
    """Loads in a document with a supported extension."""
    supported_extensions = {
        ".pdf": PyPDFLoader,
        ".txt": TextLoader,
        ".epub": EpubReader,
        ".docx": UnstructuredWordDocumentLoader,
        ".doc": UnstructuredWordDocumentLoader,
    }

def load_document(temp_filepath: str) -> list[Document]:
    """Load a file and return it as a list of documents.

    Doesn't handle a lot of errors at the moment.
    """
    ext = pathlib.Path(temp_filepath).suffix.lower()
    loader = DocumentLoader.supported_extensions.get(ext)
    if not loader:
        raise DocumentLoaderException(
            f"Invalid extension type {ext}, cannot load this type of file"
        )

    loaded = loader(temp_filepath)
    docs = loaded.load()
    logging.info(docs)
    return docs

In chat_with_documents.py:

Upgrade libraries to avoid deprecation warnings
Change SimpleSequentialChain for SequentialChain due to errors on the number of inputs required by the former
Parameterize the output_key and input variables so that they are aligned with the chain (FLARE vs normal)

"""Chat with retrieval and embeddings."""
import logging
import os
import tempfile

from langchain.chains import (
    ConversationalRetrievalChain,
    FlareChain,
    OpenAIModerationChain,
    SequentialChain,
)
from langchain.chains.base import Chain
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import EmbeddingsFilter
from langchain.schema import BaseRetriever, Document
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import DocArrayInMemorySearch
from langchain_openai.chat_models import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings

from chat_with_retrieval.utils import MEMORY, load_document
from config import set_environment

logging.basicConfig(encoding="utf-8", level=logging.INFO)
LOGGER = logging.getLogger()
set_environment()

# Setup LLM and QA chain; set temperature low to keep hallucinations in check
LLM = ChatOpenAI(
    model_name="gpt-3.5-turbo", temperature=0, streaming=True
)

def configure_retriever(
        docs: list[Document],
        use_compression: bool = False
) -> BaseRetriever:
    """Retriever to use."""
    # Split each document documents:
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=200)
    splits = text_splitter.split_documents(docs)

    # Create embeddings and store in vectordb:
    embeddings = OpenAIEmbeddings()
    # alternatively: HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
    # Create vectordb with single call to embedding model for texts:
    vectordb = DocArrayInMemorySearch.from_documents(splits, embeddings)
    retriever = vectordb.as_retriever(
        search_type="mmr", search_kwargs={
            "k": 5,
            "fetch_k": 7,
            "include_metadata": True
        },
    )
    if not use_compression:
        return retriever

    embeddings_filter = EmbeddingsFilter(
        embeddings=embeddings, similarity_threshold=0.2
    )
    return ContextualCompressionRetriever(
        base_compressor=embeddings_filter,
        base_retriever=retriever,
    )

def configure_chain(retriever: BaseRetriever, use_flare: bool = True) -> Chain:
    """Configure chain with a retriever.

    Passing in a max_tokens_limit amount automatically
    truncates the tokens when prompting your llm!
    """
    output_key = 'response' if use_flare else 'answer'
    MEMORY.output_key = output_key
    params = dict(
        llm=LLM,
        retriever=retriever,
        memory=MEMORY,
        verbose=True,
        max_tokens_limit=4000,
    )
    if use_flare:
        # different set of parameters and init
        # unfortunately, have to use "protected" class
        return FlareChain.from_llm(
            **params
        )
    return ConversationalRetrievalChain.from_llm(
        **params
    )

def configure_retrieval_chain(
        uploaded_files,
        use_compression: bool = False,
        use_flare: bool = False,
        use_moderation: bool = False
) -> Chain:
    """Read documents, configure retriever, and the chain."""
    docs = []
    temp_dir = tempfile.TemporaryDirectory()
    for file in uploaded_files:
        temp_filepath = os.path.join(temp_dir.name, file.name)
        with open(temp_filepath, "wb") as f:
            f.write(file.getvalue())
        docs.extend(load_document(temp_filepath))

    retriever = configure_retriever(docs=docs, use_compression=use_compression)
    chain = configure_chain(retriever=retriever, use_flare=use_flare)
    if not use_moderation:
        return chain

    input_variables = ["user_input"] if use_flare else ["chat_history", "question"]
    moderation_input = "response" if use_flare else "answer"
    moderation_chain = OpenAIModerationChain(input_key=moderation_input)
    return SequentialChain(chains=[chain, moderation_chain],
                           input_variables=input_variables)

In app.py:

Parameterize the params provided to the chain depending on use_flare
Change .run() to .invoke() (and the respective inputs) to avoid deprecation warnings
Extract the actual answer from the response of the chain (.invoke() returns and object with question, context and answer)

"""Document loading functionality.

Run like this:
> PYTHONPATH=. streamlit run chat_with_retrieval/chat_with_documents.py
"""
import logging

import streamlit as st
from streamlit.external.langchain import StreamlitCallbackHandler

from chat_with_retrieval.chat_with_documents import configure_retrieval_chain
from chat_with_retrieval.utils import MEMORY, DocumentLoader

logging.basicConfig(encoding="utf-8", level=logging.INFO)
LOGGER = logging.getLogger()

st.set_page_config(page_title="LangChain: Chat with Documents", page_icon="🦜")
st.title("🦜 LangChain: Chat with Documents")

uploaded_files = st.sidebar.file_uploader(
    label="Upload files",
    type=list(DocumentLoader.supported_extensions.keys()),
    accept_multiple_files=True
)
if not uploaded_files:
    st.info("Please upload documents to continue.")
    st.stop()

# use compression by default:
use_compression = st.checkbox("compression", value=False)
use_flare = st.checkbox("flare", value=False)
use_moderation = st.checkbox("moderation", value=False)

CONV_CHAIN = configure_retrieval_chain(
    uploaded_files,
    use_compression=use_compression,
    use_flare=use_flare,
    use_moderation=use_moderation
)

if st.sidebar.button("Clear message history"):
    MEMORY.chat_memory.clear()

avatars = {"human": "user", "ai": "assistant"}

if  len(MEMORY.chat_memory.messages) == 0:
    st.chat_message("assistant").markdown("Ask me anything!")

for msg in MEMORY.chat_memory.messages:
    st.chat_message(avatars[msg.type]).write(msg.content)

assistant = st.chat_message("assistant")
if user_query := st.chat_input(placeholder="Give me 3 keywords for what you have right now"):
    st.chat_message("user").write(user_query)
    container = st.empty()
    stream_handler = StreamlitCallbackHandler(container)
    with st.chat_message("assistant"):
        params = {
            "question": user_query,
            "chat_history": MEMORY.chat_memory.messages
        }
        if use_flare:
            params = {"user_input": user_query}
        config = {'callbacks': [stream_handler]}
        response = CONV_CHAIN.invoke(input=params, config=config)
        output_key = 'response' if use_flare else 'answer'

        # Display the response from the chatbot
        if response:
            container.markdown(response[output_key])

@CSalle that's really cool! (sorry, for only getting back to you now). I'll test this on langchain 0.0.284 soon.
I've started another branch that's called softupdate and runs on a newer version of LangChain (0.1.13). I've only updated a few notebooks so far (chapter 3), but if you like you could create a PR against that branch.

Done!

@mrchaos it took me a while, but I've finally started pushing these changes to main 2b886bf029f9a64486b6b461de208ef5564d227f. It's also on the softupdate branch.


Allow tracing crates other than the current crate

Due to the hardcoded env_filter here https://github.com/dbrgn/tracing-test/blob/main/tracing-test-macro/src/lib.rs#L67 it's only possible to trace logs from within the same crate as the test - would be nice to be able to specify additional crates/modules to include if the logs you're looking for are from dependency crates.

Took a quick stab at it here but ran into the env filter actually being global in the current setup, so you can't set different ones for different tests, hopefully this is still a helpful start https://github.com/reem/tracing-test/tree/add-configurable-env-filter

Hm, yeah, this would be nice to have. However, I don't know how to get around that set_global_default approach if we want to support logs from background threads.
Maybe as a workaround a no-env-filter cargo feature could be added, which disables the filtering globally?

Regarding the set_global_default did you take a look at with_test_writer()? I might misunderstand it and what this crate does but it allows me to set a custom filter function that I use to disallow/allow modules and crates. The procedural macro could generate the filter function from some attributes. I did not fully understand the issue with threads in the README so I might be completely off here.
fn setup_tracing() {
    use tracing_subscriber::{filter, prelude::*};

    let filter_modules = filter::filter_fn(|metadata| {
        if let Some(module) = metadata.module_path() {
            !module.contains("tarp")
        } else {
            true
        }
    });
    let fmt = tracing_subscriber::fmt::layer()
        .pretty()
        .with_test_writer();

    let _ignore_err = tracing_subscriber::registry()
        .with(fmt)
        .with(filter::LevelFilter::INFO)
        .with(filter_modules)
        .try_init();
}

@dvdsk IIRC the subscriber registration is thread-local, meaning that if you spawn a task that runs on another thread (or if you log from a classic thread), the logs will not be captured. I don't think this has changed lately, right?

I've just tripped over this.

Maybe as a workaround a no-env-filter cargo feature could be added, which disables the filtering globally?

FWIW I'd be happy to opt in to that approach for my integration test crate.
Alternatively could logs_with_scope_contain be used to address any further filtering that a test needs, e.g. capture everything via the filter but then have logs_assert take a scope (provided by the macro generated wrapper function) but provide another API which asserts over all logs?
I also noticed that [tracing_fmt::Builder]https://docs.rs/tracing-fmt/0.0.1-alpha.3/tracing_fmt/struct.Builder.html) has methods  with_filter_reloading, reload_handler and with_filter etc which might allow something more dynamic compared with the env_filter approach.

Alternatively could logs_with_scope_contain be used to address any further filtering that a test needs, e.g. capture everything via the filter but then have logs_assert take a scope (provided by the macro generated wrapper function) but provide another API which asserts over all logs?

Hmm, that could work! Would you be willing to give it a try?
(Please also check out the discussion in #12.)

I also noticed that tracing_fmt::Builder has methods  with_filter_reloading, reload_handler and with_filter etc which might allow something more dynamic compared with the env_filter approach.

With a global subscriber, that will only work if your tests do not run in parallel. Since cargo test runs tests in parallel by default, I wouldn't be happy about such a limitation.

@dvdsk IIRC the subscriber registration is thread-local, meaning that if you spawn a task that runs on another thread (or if you log from a classic thread), the logs will not be captured. I don't think this has changed lately, right?

You are absolutely right, I ran into that :)

I'm following this discussion, as the same issue prevents me to use tracing-test for integration tests within my same project as well. Since technically, each integration test is compiled as its own crate, most of the logs I'd want to keep and assert upon are filtered out.

I'm following this discussion, as the same issue prevents me to use tracing-test for integration tests within my same project as well. Each integration test is compiled as its own crate, therefore most of the logs I'd want to keep and assert upon are filtered out.

+1

An implementation of a no-env-filter feature is implemented in #16:

Per-crate Filtering
By default, tracing-test sets an env filter that filters out all logs
except the ones from your crate (equivalent to
RUST_LOG=<your_crate>=trace). If you need to capture logs from other crates
as well, you can turn off this log filtering globally by enabling the
no-env-filter Cargo feature:
tracing-test = { version = "0.1", features = ["no-env-filter"] }

Note that this will result in all logs from all your dependencies being
captured! This means that the logs_contain function may become less
useful, and you might need to use logs_assert instead, with your own
custom filtering logic.

Could some of the people here please test this?
-tracing-test = "0.2.1"
+tracing-test = { git = "https://github.com/dbrgn/tracing-test/", branch = "no-env-filter", features = ["no-env-filter"] }


Approximate closed-form rebalancing for faster updates and more meaningful half-lives

@fasiha, I dare say, I think you're going to like this!
Overview
I've been working on a pure-Python implementation of the ebisu algorithm and I was not satisfied with the current approach to determining a new half-life during updates (that being 'just use the old half-life, unless the Beta model gets too unbalanced, then 'rebalance' it using a coarse grid search over halflives').
I came up with a different approach which turns out, from my initial experiments, to be simpler, work faster, and actually lead to more accurate/meaningful half-life updates than the current rebalancing strategy (see also #31).
Key idea
They key idea is as follows:

As you know (indeed as you invented), the half-life parameter is the time elapsed at which the recall probability is distributed according to a beta distribution with the other two parameters.

Thus this 'half-life' parameter is at its most meaningful when that beta distribution is roughly centered at p=0.5, where we will have something approximating exponential decay with half-life equal to the parameter we have been calling 'half-life'.

You have shown that in this case the time-decaying beta distribution's mean does not undergo exponential decay exactly, but that it's pretty close. If we were to (approximately) model the probability of recall's mean over time as exponential decay with some half-life, we would get an (approximate) closed form equation for that half-life in terms of the mean at any other given time.
Let P_recall@t be the random variable representing the probability of recall at elapsed time t. Then:
E[P_recall@t] =approx. 2 ^ {-t / λ}
where λ is the half-life of the exponential decay. We can invert this equation to get:
λ =approx. - t / log_2 E[P_recall@t].

My approach is essentially to use such an equation on the analytic posteriors to determine a new value for the half-life parameter during every update.

Algorithm
The resulting update algorithm works as follows. If my understanding of the ebisu algorithm is correct, the only change from the current algorithm is in step 4.

We have the prior over recall probability at time λ_old, distributed according to a Beta distribution with parameters α_old, β_old. I'll denote this prior P_recall@λ_old ~ Beta(α_old, β_old).
Because the quiz takes place at time t, we shift this prior through time before the update, to prior P_recall@t ~ GB1(...).
We perform the update, computing posterior P_recall@t ~ (that complex distribution with analytical moments).
We want to find a suitable half-life for this posterior:

Move the posterior back to time λ_old, giving posterior P_recall@λ_old ~ (another complex distribution with analytical moments).
Compute the first moment of this posterior at λ_old, giving us E[postr P_recall@λ_old].
The equation mentioned above approximately governs the approximately exponential decay of this posterior's mean. In particular, E[P_recall@λ_old] =approx. 2 ^ {-λ_old / λ_new} where λ_new can be interpreted as the 'effective half-life' of the posterior and will be our choice for the new λ parameter.
Plug the posterior's first moment and λ_old into this equation to calculate λ_new =def. - λ_old / log_2 E[postr P_recall@λ_old].

With our suitable new half-life chosen, move the posterior to λ_new, posterior P_recall@λ_old ~ (another complex distribution with analytical moments), which should be approximately central.
Moment match a Beta distribution to give us posterior P_recall@λ_new ~approx. Beta(α_new, β_new), which we take as our new memory model.

Note: some of those steps are conceptual rather than computational; pseudocode illustrating the computational steps is below (all in log-space, of course):
def updateRecall(result r, time elapsed t, prior model parameters θ):
    # Steps 1--2 are captured by assumption within the parameters
    _, _, λ_old = θ

    # Steps 3 and 4:
    # calculate the posterior after update at time t, shifted back to time λ_old
    postr_λ_old = analytic_posterior(r, t, λ_old, θ)
    # use this posterior mean and the exponential decay approximation as
    # a heuristic to determine a new half-life
    ln_μ_λ_old = postr_λ_old.ln_moment(1) # ln here denotes natural logarithm
    λ_new = - λ_old * ln(2) / ln_μ_λ_old

    # Step 5. Compute the moments of the posterior at time λ_new
    postr_λ_new = analytic_posterior(r, t, λ_new, model)
    ln_m1 = postr_λ_new.ln_moment(1)
    ln_m2 = postr_λ_new.ln_moment(2)
    mean  = exp(ln_m1)
    var   = exp(ln_m2) - exp(2*ln_m1)
    # Step 6. match with a beta distribution to fit our new model
    α_new, β_new = beta_match_moments(mean, var)
    return (α_new, β_new, λ_new)

Results
I have experimented by performing these Bernoulli updates at 0.0001, 0.01, 1, 100, and 10000 half-lives, for both failed (successes=0) and passed (successes=1) trials. The initial half-life was exactly 1 (so t = δ).
I tried the following four update schemes for comparison:

λ_new = λ_old (no rebalance): Do not attempt to rebalance, just force fit a Beta distribution to the posterior, no matter how unequal alpha and beta are.
λ_new = λ_old (no rebalance)
result  delta    new alpha       new beta      new half-life
fail    1e-04        1.665871    3.093668      1
fail    1e-02        1.670583    3.093355      1
fail    1e+00        2.000000    3.000000      1
fail    1e+02        2.004093    2.006296      1
fail    1e+04        2.000000    2.000001      1
pass    1e-04        2.000100    2.000000      1
pass    1e-02        2.010000    2.000000      1
pass    1e+00        3.000000    2.000000      1
pass    1e+02      102.000000    2.000000      1
pass    1e+04    10012.410073    2.002081      1

OK, no question, this strategy is Really Bad, obviously. The half-life parameter loses all interpretability and, as I think you mentioned elsewhere, moment-matching would probably be pretty inaccurate for such extremely skewed posteriors.

ebisu (AUTO rebalance): The current implentation: Do (1) unless alpha and beta differ by a factor of two or more, then use coarse grid-search to find a new half-life.
ebisu (AUTO rebalance)
result  delta    new alpha       new beta      new half-life
fail    1e-04        1.665871    3.093668      1
fail    1e-02        1.670583    3.093355      1
fail    1e+00        2.000000    3.000000      1
fail    1e+02        2.004093    2.006296      1
fail    1e+04        2.000000    2.000001      1
pass    1e-04        2.000100    2.000000      1
pass    1e-02        2.010000    2.000000      1
pass    1e+00        3.000000    2.000000      1
pass    1e+02        1.328794    2.075719     61.566292
pass    1e+04        2.589229    2.032667   3361.405627

Behaving as expected. Interestingly, there is never a need to rebalance for fail trials even for extremely quick fails.

ALWAYS ebisu-rebalance: Always use coarse grid-search to find a new half-life.
ALWAYS ebisu-rebalance
result  delta    new alpha       new beta      new half-life
fail    1e-04        1.430187    3.132588      1.127626
fail    1e-02        1.434299    3.132170      1.127626
fail    1e+00        1.725811    3.034261      1.127626
fail    1e+02        1.749764    2.017428      1.127626
fail    1e+04        1.745926    2.010629      1.127626
pass    1e-04        1.746014    2.010628      1.127626
pass    1e-02        1.754725    2.010569      1.127626
pass    1e+00        2.627134    2.006456      1.127626
pass    1e+02        1.328794    2.075719     61.566292
pass    1e+04        2.589229    2.032667   3361.405627

Also behaving as expected. Interestingly, the grid search finds an increased half-life parameter as the best fit even for failed trials. Of course, the effective 'half-life' determining scheduling probably still goes down, and this is probably captured through appropriate changes in alpha and beta.

always APPROX. rebalance: Always use the approximation discussed above to find a new half-life.
always APPROX. rebalance
result  delta    new alpha       new beta      new half-life
fail    1e-04        2.763985    2.994630      0.660264
fail    1e-02        2.765495    2.994940      0.661462
fail    1e+00        2.790686    2.936193      0.756471
fail    1e+02        2.005878    2.006227      0.999208
fail    1e+04        2.000001    2.000001      1.000000
pass    1e-04        2.000019    2.000003      1.000036
pass    1e-02        2.001878    2.000299      1.003606
pass    1e+00        2.135892    2.018352      1.356915
pass    1e+02        2.487732    2.034483     35.695958
pass    1e+04        2.501217    2.034258   3466.775981

This behaviour is, I think, really neat! So we see that for the cases where the existing ebisu algorithm (above) decided to rescale the half-life, this approximate method got a similar value (e.g. 3361 v. 3466, 61 v. 35, about 1 v. about 1). In a few notable cases, I think it did a 'better' job, especially in that the half-life actually goes down for failed trials. Perhaps most importantly, the changes to the alpha and beta parameters at the new half-life are pretty contained, i.e. the beta distribution has stayed very well balanced. More on this in the following plot.

Finally, here's a plot of some of the posterior Beta distributions at their new half-lives after these updates, for the various algorithms. Of course, the no-rebalancing scheme is useless, and the ebisu rebalancing scheme works pretty well. But here you can see visually how the new approximate balancing scheme looks even a little more centered at its half-life than the current ebisu rebalancing scheme.

Remaining issues

So far, I have only implemented this for Bernoulli quizzes, because that's all my pure-Python port aims to implement (because my memory app doesn't offer any Binomial trials with total > 1). However, I think the math would go through regardless and so this approach could easily be incorporated into ebisu proper in principle.

I haven't extensively tested the approach, it may be possible that the exponential approximation is pretty dramatically off outside of the range of values I tested, and there it may lead to bad updates. I was actually hoping that with your experience testing ebisu algorithms you might be able to (help) confirm whether this algorithm works over a broader range of situations.

The result of the algorithm is still not completely centered, and so perhaps over many updates it would become unbalanced and need a more sophisticated rebalancing operation. Or, for reasons deeper than what I have considered, this approach might be kind of self-stabilising. Anyway, either way, to be safe it would be easy to use as a default strategy followed by a more sophisticated balancing strategy if the alpha and beta get too out of kilter.

In case you are interested, I do have code to back this up and I am in principle willing to share it all in the public domain, but it's not quite ready for sharing yet. Anyway you can probably find some of it if you look hard enough around my own github.

First, thank you so much for spending so much time and attention on Ebisu, on coming up with this approach, and for writing it up in such detail! I am very grateful!!
Second—yes! I like this!!! I am confident you have found a much better alternative to the rebalancing that we currently do—we can replace the very ad hoc check for rebalance (0.5 < α / β < 2) and the coarse grid search with a very reasonable and self-explanatory alternative.
And as you note, it will give us a cheap and good approximation to start the fine search for the true halflife (that gives us posterior mean of 0.5) to properly do #31.
I'm also confident it will be straightforward to incorporate this approach with the binomial quiz case.
Are you interested in a version of Ebisu that does #31 (always rebalances), using this approach? If so, I can try to focus on that plus some other outstanding issues that are mostly done and release a new minor version—alas due to life commitments, I'm not sure when I'll be able to give this the time it needs. But it sounds like you have a version that is working for you, and while, yes, it may be possible that we need a fallback in case this algebraic approach to approximating the halflife fails, I think barring pathological cases it should work just fine.
Again, many, many thanks for your hard work!!!

First, thank you so much for spending so much time and attention on Ebisu, on coming up with this approach, and for writing it up in such detail! I am very grateful!!
Second—yes! I like this!!! I am confident you have found a much better alternative to the rebalancing that we currently do—we can replace the very ad hoc check for rebalance (0.5 < α / β < 2) and the coarse grid search with a very reasonable and self-explanatory alternative.
And as you note, it will give us a cheap and good approximation to start the fine search for the true halflife (that gives us posterior mean of 0.5) to properly do #31.
I'm also confident it will be straightforward to incorporate this approach with the binomial quiz case.
Are you interested in a version of Ebisu that does #31 (always rebalances), using this approach? If so, I can try to focus on that plus some other outstanding issues that are mostly done and release a new minor version—alas due to life commitments, I'm not sure when I'll be able to give this the time it needs. But it sounds like you have a version that is working for you, and while, yes, it may be possible that we need a fallback in case this algebraic approach to approximating the halflife fails, I think barring pathological cases it should work just fine.
Again, many, many thanks for your hard work!!!

Right, I have my own implementation, which I will continue to use for my memory exercises. So there's no pressure from me to get this into a new minor version. Please consider this issue a 'feature suggestion' rather than an 'issue' or urgent feature request.
In the mean time, I'll continue to use (my implementation of) the ebisu algorithm for my memory exercises and I can update you once I have more data on how this heuristic updating scheme performs, or if it destroys my memory models, or if I come across any other suggestions.

Right, I have my own implementation, which I will continue to use for my memory exercises. So there's no pressure from me to get this into a new minor version. Please consider this issue a 'feature suggestion' rather than an 'issue' or urgent feature request.
In the mean time, I'll continue to use (my implementation of) the ebisu algorithm for my memory exercises and I can update you once I have more data on how this heuristic updating scheme performs, or if it destroys my memory models, or if I come across any other suggestions.


issue when inserting into concurrent map

Hi,
I have a seemingly simple test program which hangs on my machine under certain circumstances:
#include <junction/ConcurrentMap_Leapfrog.h>
#include <iostream>

struct Value
{
    int v;
};

using ConcurrentMap = junction::ConcurrentMap_Leapfrog<int,Value*>;

int main()
{
    ConcurrentMap cmap(10000);
    Value value{ 0 };

    //for (int i = 0; i < 1000000; ++i) // locks in 158th assign
    for (int i = 999999; i >= 0; --i) // works as expected
    {
        cmap.assign(i, &value);
        std::cout << "inserted " << i << "\n";
    }

    std::cout << "done\n";
}

As you can see from the comment around the for loops, one case works as expected (completes without error) while the other one hangs.  I am at a loss to explain why.  Is this a bug, or a known issue?  Or am I doing something stupid?
Thanks for your time.

Update: yes I know I was missing a zero in the call to the constructor.  When I change that to 1000000, it gets a little bit further along, but still hangs.  If I don't supply a size to the constructor, then both loops work fine.

Hi mdevico,
The initial table size (passed to the constructor) must be a power of 2. You should see the assert if you run in Debug mode.
TURF_ASSERT(turf::util::isPowerOf2(tableSize));


Support generating bindings from dependent crates

Can csbindgen look at a crates's dependencies when generating bindings? Let's imagine
Crate dll:
pub use api_types::Vec2;

fn function(x: *const Vec2) {}

Crate types:
#[repr(C)]
pub struct Vec2 {
    pub x: f32,
    pub y: f32,
}

Currently it looks like it doesn't search dependencies for types to output. If we run csbindgen in dll/build.rs, it won't include types defined in types.

We do not plan to add dependency analysis currently (or in the future).
Please call input_bindgen_file multiple times and manually add the necessary dependency files as needed.

Yeah, thanks. I managed to set up the structure such that it somehow works.

@yanchith Could you please share some sample how you achieved this.
I want to create a C# binding for existing rust lib. I would just use as a dependecy and reference it for build.
@neuecc  Do you have any exmaple for similar?
I appreciate you help.
 I'm not familliar with rust ecosystem.)

@pherbel I literally just applied the suggestion, and called input_bindgen_file multiple times on multiple files.

I want to create a C# binding for existing rust lib. I would just use as a dependecy and reference it for build.

Unless the library was specifically structured beforehand to gel well with bindings generators, you might run into trouble. I can't share the codebase I am working on, but my loose inspiration was the wgpu crate from around 3-4 years ago. They used to have (maybe still do, IDK) a very clean separation of what is visible from C and what isn't. You can still see parts of it in https://github.com/gfx-rs/wgpu/blob/trunk/wgpu-types/src/lib.rs, where they define the C-visible types.


Add support for multiple #import in the same file in schema files

Related to #38
import was not working when having multiple #import statements in the same file.
Elements defined directly in this file were added multiple times ( one for each import )
This is a proposition of fix this issue. I tried to keep the same spirit of existing code and test.

Good catch, and great work! Thank you. :)


SG Support says the devs are in here... Where are you devs!!

Issue Summary
Customer support from SG is, quite frankly, shocking and seriously lacking!!

@Robin-the-Frog We're taking steps to improve response times here. Is there an issue in particular that you need help with?

Just the fact that my comment was made on 3rd November and it's taken until 17th March for a response says it all!
I had posted about 3 or 4 different questions around that time, none got any response for months. So long, that I worked them all out for myself by pure trial and error, and in fact I even helped some other posters with their questions from what I'd learnt myself...

Understood. Appreciate your help in assisting others in the community. As I said, we're taking steps to improve response times while at the same time working through the large backlog of open issues (over 1k in total). It's a slow process, but we're committed to it.


Restructure Clients and Add ScienceBase to the Clients

We have FDSN clients working for the most part, now we should look at adding other clients.  One in particular that has MHT5 files already is USGS ScienceBase.  We can use sciencebasepy as the tool to get data and data availability.  This is not a data services like IRIS, its an file drive.  So you can only pull the files it has on it and not parts of files like you can with IRIS, namely you can't request a time period.
If we do this we should restructure clients to be more plugin friendly.  For instance have a FDSN module and a ScienceBase module that make_mth5 can load.  This will make it easier to extent to other clients
|-> clients
|-> make_mth5
|-> fdsn
|-> science_base
|-> others

@kkappler and @timronan  what do you guys think?  This would sort of go along with Karl's intuition to split up some of the functions in make_mth5.  I'll create a branch clients that we can work on.

@kujaku11 this sounds good. Would the structure be similar to:
|-> clients
|--> make_mth5
|----> fdsn
|----> science_base
|----> others

So a switch could be added into  make_mth5 which would direct which client is called. It seems like this would allow us to reuse a lot of the logic in Make_MTH5 and make client specific plugins as you mention above.

Done with PR #139


Fixed link color contrast on debug docs page.

Before:

After:

Coverage remained the same at 67.089% when pulling 101992503a9ed9e34e4f786aaea3e4a8fe98e2dc on toanzian/docs-color into ea856fd3a77c9c2a92d4d777393cf190051694c1 on master.


Long grass not spawning on grass blocks when using bonemeal

Hi,
Thanks for the great mod, my friends and I are loving it!
I'm not sure if this is expected or not, but when we use bonemeal on grass blocks, no long grass is spawning.
We are running Minecraft/Fabric 1.20.4 with mod version 1.9.0 with entities.
Many thanks,
George

This is definitely not intended and have a feeling I might know where the issue lies.
Will try to fix it as soon as possible, thanks for letting me know ^-^

uploaded a new version that should have fixed the issue


Add full support for fuzzing to maelstrom-go-test

Some considerations include:
We could parse out the test data for the fuzz tests and run each fuzz test as a separate job. This won't work when the fuzz test seeds are specified with F.Add.
We'll need to accept command-line-options or configuration values to set the minimum and maximum amount of time to spend on a fuzz test (or minimum and maximum number of iterations).
We need to figure out how instrumentation works. The go documentation says:

On supported platforms, 'go test' compiles the test executable with fuzzing coverage instrumentation. The fuzzing engine uses that instrumentation to find and cache inputs that expand coverage, increasing the likelihood of finding bugs.

We'll need a way to gather seeds for failure, report the failures they induced, and save the seeds.
At its most developed, one could imagine having the test coordinator run an arbitrary number of jobs, where each job did some amount of fuzzing based off of a seed. The test coordinator would look at the results it was getting back and create more jobs dynamically. Basically, wouldn't it be cool if you had a cluster of 1000 cores if you ask maelstrom to run a fuzz test for 10 seconds across the whole cluster, and get a meaningful amount of coverage?
We'd need to figure out how to run fuzz tests. This is a good example of the problem of having "open ended tests" of "test templates". When somebody just runs all of the tests, what do they want to happen with these templates? What values do they want? Anyway, that's a larger question. Maybe in the short term we'd just accept command-line options that would only run fuzz tests.
See: https://go.dev/doc/security/fuzz/
See: https://pkg.go.dev/testing#hdr-Fuzzing

Part of #319


Uses a sane default for pysftp_conn_kwargs

Description
pysftp_conn_kwargs defaults to None, which causes a TypeError when passed
with ** for the kwargs of pysftp.connect. This fixes replaces None with {} when storing pysftp_conn_kwargs in the RemoteFileSystem.
Motivation and Context
I found that since updating to the latest luigi yesterday, my sftp tasks have stopped working. There would be an error like TypeError: type object argument after ** must be a mapping, not NoneType in the luigi ftp library. I traced it down to the default value for the new optional argument pysftp_conn_kwargs in RemoteFileSystem. By storing {} when None is passed, the error is fixed.
Have you tested this? If so, how?
I ran a previously failing job with this code and it succeeded.

eesh, that's embarrassing.  good catch. +1.
I guess this is what you get for contributing code and never using the default.

No worries, it was a pretty easy fix. Unfortunately, I think this made it into the 2.3.0 release, so we'll probably want to bump to 2.3.1. This can be a fairly costly bug, as google recently started requiring sftp for some ad-related uploads.

correct me if I'm wrong, but I was looking at the tests today and it seemed like luigi doesn't have any tests in the suite for the SFTP component of RemoteTarget? I think I was looking at ftp_test and there were only FTP tests in there. That might be exacerbating the issue.

@daveFNbuck. Do you think I should merge this immediately and release 2.3.1?

@Tarrasch I think so. Loss of sftp can be pretty critical.
@petergaultney agreed, we should add some sftp tests. I was just blindly using it until today :)


MiAIRR Compliance Flag

A small change which introduces a flag to check for MiAIRR Compliance in the data (defaults to True).
If True, the current behaviour of validate_airr is followed.
If False, the data structure is checked by the MiAIRR and nullable checks are ignored.
There is a larger piece of work to unify the nullable check and the miarr check into one piece of logic as there is duplication. Please correct me if I'm wrong but miairr: essential if and only if nullable: false. As miairr is more developed than the nullable check, nullable could be deprecated in favour of only having and checking the miairr field.
I'm happy to implement this in the future if it is something that the AIRR community would find valuable.

Please correct me if I'm wrong but miairr: essential if and only if nullable: false.

Not exactly, while miairr: essential implies nullable: false, the opposite implication is not true, there are fields with nullable: false that are not part of MiAIRR.

As miairr is more developed than the nullable check, nullable could be deprecated in favour of only having and checking the miairr field.

That would actually complicate validation for APIs like the ADC API. Because nullable is a standard keyword in OpenAPI 3, standard validation routines can insure that nullable flag is being enforced properly in the data. However, miairr is a custom extension and thus cannot be validated without custom code.
The idea of a MiAIRR compliance flag might be useful, but not necessarily for miairr: essential as standard schema keywords like required and nullable can enforce that. It might be more useful for miairr: important

Please correct me if I'm wrong but miairr: essential if and only if nullable: false.

Not exactly, while miairr: essential implies nullable: false, the opposite implication is not true, there are fields with nullable: false that are not part of MiAIRR.

As miairr is more developed than the nullable check, nullable could be deprecated in favour of only having and checking the miairr field.

That would actually complicate validation for APIs like the ADC API. Because nullable is a standard keyword in OpenAPI 3, standard validation routines can insure that nullable flag is being enforced properly in the data. However, miairr is a custom extension and thus cannot be validated without custom code.
The idea of a MiAIRR compliance flag might be useful, but not necessarily for miairr: essential as standard schema keywords like required and nullable can enforce that. It might be more useful for miairr: important

Thanks for the clarification.
In that case, I think the long term aim after this PR would be a miairr_compliance_flag which is set to essential by default, but also supports important and defined.

I think a more useful design is to quantify how compliant some data is to MiAIRR, specifically for the different levels. By quantify, I mean count up the number of fields in the schema with (say) miairr: important then in the data, count up how many of those fields have data in them, then output the %.
In terms of implementation, this wouldn't be an additional flag on the various load functions. Instead it would be a standard-alone function, like validate_object, where you pass the data and what compliance level to check as parameters. It would be simpler function because it only needs to recurse through the data structure and check MiAIRR.
Then to make it a bit more useful, update airr-tools program to accept a "miairr compliance" option to run on a data file.

@jday1 Is this closer to your intention of a "MiAIRR Compliance Flag"?

@schristley are you good with this as is?

@jday1 This branch is very old and cannot be merged as is. Your changes look ok to me, can you create a new branch/PR based upon recent master? That would insure there are no conflicts with all of the recent v2 work and we can merge easily.

@jday1 This branch is very old and cannot be merged as is. Your changes look ok to me, can you create a new branch/PR based upon recent master? That would insure there are no conflicts with all of the recent v2 work and we can merge easily.

Thanks @schristley. I have merged the most recent version of master into this branch and the checks all pass. Please merge when suitable.


Add custom API key support in AI configuration

This unlocks support for most open api compatible providers online

Hey @swiftugandan , thanks for the contribution. Given that someone added gemini since you wrote this there are conflicts. Mind rebasing and fixing the merge conflicts?

@nichochar rebase done

Code LGTM, will test this locally and confirm it works, and if it does merge. Thanks!


fix dummy display in safari

#fix# demo display issue in safari.

Changes Unknown when pulling 38d1b687573e15c73da0f2c9f0f54766f860b7c0 on quincyle:master into ** on ciena-frost:master**.

Changes Unknown when pulling 38d1b687573e15c73da0f2c9f0f54766f860b7c0 on quincyle:master into ** on ciena-frost:master**.

Approved

Approved


Use intake

Pull Request Checklist:

[x] This PR addresses an already opened issue (for bug fixes / features)

This PR is related to issue https://github.com/roocs/34e-mngmt/issues/93

[x] Tests for the changes have been added (for bug fixes / features)
[x] Documentation has been added / updated (for bug fixes / features)
[x] HISTORY.rst has been updated (with summary of main changes)
[ ] I have added my relevant user information to AUTHORS.md

What kind of change does this PR introduce?:

Inventory maker has been removed and replaced by intake catalog maker which writes a csv file with the dataset entries and a yaml description file.
It includes file sizes and start and end times. It also includes latitude and longitude as separate columns - these could be made into one bbox column if preferred.
I've taken away array_dims and array_shape as they weren't in https://github.com/cp4cds/c3s_34g_manifests/blob/master/intake/catalogs/c3s-cmip6_v20210313.csv.gz but they can be put back in.
In Carsten's example, there was an entry for each file - so I've adjusted the scanner so that it loops through each file and uses the file path as its key in the records dictionary, rather than the dataset id. There may be a more efficient way to do this?
In etc/roocs.ini the option use_inventory has been replaced by use_catalog and the inventory maker options have been replaced with equivalent catalog options. However, the option to include file paths or not no longer exists.
I've added tests to check the catalog maker works as expected.

Does this PR introduce a breaking change?:
yes - pandas is a new dependency

Other information:
As mentioned - it may be better for this catalog builder to be moved elsewhere.

* **Other information:**
  As mentioned - it may be better for this catalog builder to be moved elsewhere.

yes 😄  either a new package or part of the catalog? Can be decided in next step.

@ellesmith88 looks good to me 👍  ... just comments ... but could all be done in another PR.

@ellesmith88 thanks for putting this PR together. There is a lot of work in there :-)
I have read through the changes quickly (sorry - short on time) and I wasn't sure whether the intake records are covering both files and datasets. We need both because:

The inventory-like structure is required for the CDS Team (i.e. what was our YAML inventory).
The rook director uses the file-based records in Intake in order to check the datetime range for each file.

Is that covered in this PR, or is this the first stage that only deals with (1)? Thanks

Is that covered in this PR, or is this the first stage that only deals with (1)? Thanks

@agstephens The csv catalog has an entry for each file but for every file it also has the associated the ds_id, so there's several entries for one dataset id if there's more than one file:

It's based on the catalog Carsten created - https://github.com/cp4cds/c3s_34g_manifests/blob/master/intake/catalogs/c3s-cmip6_v20210313.csv.gz
So the information in each entry is only based on the files at the moment. So this covers (2), but I'm not sure if it covers (1). Maybe the size of the dataset is needed?
I've just thought that previously I had been providing the size of the xarray dataset - here is it the size of the file that's needed?

So the information in each entry is only based on the files at the moment. So this covers (2), but I'm not sure if it covers (1). Maybe the size of the dataset is needed?

I would also do it as it is now (2) ... collecting all infos on file level base. We have the dataset id to get the aggregation. I would use the intake catalog only as an information base. We can load the intake catalog and use pandas to query, aggregate and display the infos.
When we have the catalog available we can provide a notebook with examples how to do this. Needs to be discussed with James how they like to have the infos ... it should only need a few lines of (pandas) code to make it work.

@agstephens @cehbrecht I've updated so that there is only one catalog and we have a bbox column

I've just thought that previously I had been providing the size of the xarray dataset - here is it the size of the file that's needed?
yes ... I need the file size for the provenance info. Dataset size can be calculated from the aggregation.

yes ... I need the file size for the provenance info. Dataset size can be calculated from the aggregation.

ok 👍 Changed to file size and removed size_gb.


Add poll_interval

Fixes #2473
Special notes for your reviewer:
Thank you for reviewing
┆Issue is synchronized with this Jira Task by Unito

@kannon92 I did changes and also run tox -e docs-check
(armada38) sarthak_negi@:~/armada/third_party/airflow$ python3 -m tox -e docs-check
.pkg: _optional_hooks> python /home/sarthak_negi/armada/armada38/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /home/sarthak_negi/armada/armada38/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /home/sarthak_negi/armada/armada38/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
docs-check: install_package_deps> python -I -m pip install 'apache-airflow>=2.6.3' armada-client 'grpcio-tools>=1.46.3' 'grpcio>=1.46.3' sphinx-jekyll-builder==0.3.0 sphinx-markdown-builder==0.5.5 sphinx-toolbox==3.2.0b1 sphinx==7.0.1 'types-protobuf>=3.19.22'
docs-check: install_package> python -I -m pip install --force-reinstall --no-deps /home/sarthak_negi/armada/third_party/airflow/.tox/.tmp/package/3/armada_airflow-0.5.1.tar.gz
docs-check: commands[0] /home/sarthak_negi/armada/third_party/airflow/docs> sphinx-build -M jekyll source build
Running Sphinx v7.0.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output...
building [jekyll]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources...
looking for now-outdated files... none found
no targets are out of date.
build succeeded.
The jekyll files are in build/jekyll.
docs-check: commands[1] /home/sarthak_negi/armada/third_party/airflow/docs> bash -c 'grep -v '"'"'^date:'"'"' build/jekyll/python_airflow_operator.md > /tmp/python_airflow_operator_generated.md'
docs-check: commands[2] /home/sarthak_negi/armada/third_party/airflow/docs> diff /tmp/python_airflow_operator_generated.md ../../../docs/python_airflow_operator.md
.pkg: _exit> python /home/sarthak_negi/armada/armada38/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
docs-check: OK (62.94=setup[62.42]+cmd[0.48,0.04,0.00] seconds)
congratulations :) (63.20 seconds)
Still pr is falling could you pls give me a lead

yea it seems like something went wrong here.  https://github.com/armadaproject/armada/pull/2808
I'll aim to fix it.

yea it seems like something went wrong here. #2808
I'll aim to fix it.

So, this is the same reason for my prev PR also?

@kannon92 pls review the PR if the issue is fixed!! #2808

I fixed that issue but it looks like docs are wrong.
Please run tox -e docs and push those changes.

@kannon92 pls review the PR if the issue is fixed!! #2808

I fixed that issue but it looks like docs are wrong.
Please run tox -e docs and push those changes.

ok I will do it once again

Once this is green I think its mergeable! Thank you!

You always guided me , your welcome

@Sharpz7  could you pls review the changes !!

@kannon92  I got a success message (armada38) sarthak_negi@LAPTOP-68O9S7GK:~/armada/third_party/airflow$ python3 -m tox -e docs-check
.pkg: _optional_hooks> python /home/sarthak_negi/armada38/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /home/sarthak_negi/armada38/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /home/sarthak_negi/armada38/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
docs-check: install_package_deps> python -I -m pip install 'apache-airflow>=2.6.3' armada-client 'grpcio-tools>=1.46.3' 'grpcio>=1.46.3' sphinx-jekyll-builder==0.3.0 sphinx-markdown-builder==0.5.5 sphinx-toolbox==3.2.0b1 sphinx==7.1.2 'types-protobuf>=3.19.22'
docs-check: install_package> python -I -m pip install --force-reinstall --no-deps /home/sarthak_negi/armada/third_party/airflow/.tox/.tmp/package/3/armada_airflow-0.5.3.tar.gz
docs-check: commands[0] /home/sarthak_negi/armada/third_party/airflow/docs> sphinx-build -M jekyll source build
Running Sphinx v7.1.2
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
writing output...
building [jekyll]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources...
looking for now-outdated files... none found
no targets are out of date.
build succeeded.
The jekyll files are in build/jekyll.
docs-check: commands[1] /home/sarthak_negi/armada/third_party/airflow/docs> bash -c 'grep -v '"'"'^date:'"'"' build/jekyll/python_airflow_operator.md > /tmp/python_airflow_operator_generated.md'
docs-check: commands[2] /home/sarthak_negi/armada/third_party/airflow/docs> diff /tmp/python_airflow_operator_generated.md ../../../docs/python_airflow_operator.md
.pkg: _exit> python /home/sarthak_negi/armada38/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
docs-check: OK (49.14=setup[48.71]+cmd[0.42,0.01,0.00] seconds)
congratulations :) (49.18 seconds)

@Sharpz7 I think this pr is not merging

Yea, I lost my approval rights so it only has 1/2 required approvers.

Yea, I lost my approval rights so it only has 1/2 required approvers.

ohh


Completion for nova command

could be great to install "nova.bash_completion"

Fixed with these commands:
curl -O https://raw.githubusercontent.com/openstack/python-novaclient/master/tools/ /etc/nova.bash_completion
echo "source /etc/nova.bash_completion" >> /etc/profile


Message when a 3bot creation fails is too technical

"Please use the refresh button on the upper right to restart the chatflow"
Technically it is a chatflow but from a user perspective it is a deployment process.
I would say:" "Use the refresh button on the upper right to restart the 3bot creation"

this error is part of standard error handling in chatflowengine itself, probably can be linked to the chatflow title https://github.com/threefoldtech/js-sdk/blob/9fc257715808a9cde69daa4bcbfa204897d86a0b/jumpscale/sals/chatflows/chatflows.py#L84

branch: development
commit: f0aa7a4148b39b3ad1c8f7700454485215f2a366


Updated default.hover & default.pressed from grey030 to grey040

Updated Light-theme color tokens.
default.hover :  grey030 --> grey040
default.pressed:  grey030 --> grey040
Reason for update : grey030 is too light.
GIF shows in grey040 which has more contrast
Screenshots
Before
(Add before)
After

This changes it back to what is already in use? We don't want to to use the same color as background-alternative for the reasons listed in this PR https://github.com/MetaMask/metamask-extension/pull/16519

Decided on leaving as is because:

No accessibility issues
We don't have any other greys that we can use and don't want to add another one
The current solution prevents any merging of components into the background on hover


TypeScript support

Thanks for your project.
Could you please add type support? Currently I have to use ts-ignore
https://github.com/LibreService/my_file_type/blob/49060cb96ee1418e79ac28e8e082a3f04805487e/src/components/MyPreviewText.vue#L9-L10

Hey @eagleoflqj, sure I'll look into adding typescript support. I'll let you know when I get it working so that you can test it with your project.

Hey @eagleoflqj I just made a new release to attempt to fix the types not working. Could you try updating the package and checking if it's working now?

my_file_type$ npx vue-tsc --noEmit
src/components/MyPreviewText.vue:9:19 - error TS7016: Could not find a declaration file for module 'highlightjs-cobol'<EMAIL_ADDRESS>implicitly has an 'any' type.
  Try `npm i --save-dev @types/highlightjs-cobol` if it exists or add a new declaration (.d.ts) file containing `declare module 'highlightjs-cobol';`

9 import cobol from 'highlightjs-cobol'
                    ~~~~~~~~~~~~~~~~~~~

Found 1 error in src/components/MyPreviewText.vue:9

It seems not fixed. Maybe we need a .d.ts file?

Maybe we need a .d.ts file?

Most likely yeah. The highlight.js package has it's own .d.ts file. Could you try copying it over to the cobol package's folder?
If that fixes it I'll add that file to the repo as well.

Hey @eagleoflqj I just made another release (0.3.3) with a d.ts file. Could you check if that fixes it?

It's fixed. Thank you!


Exclude support

Thanks for porting this to Grails 3!  It's been working just fine for us.  I notice that table exclusions haven't made it yet, i.e.
grails:
    profile: web
    codegen:
        defaultPackage: test-db
    plugin:
        databasemigration:
            ignoredObjects:
                -my_excluded_table

is not honored yet when running "grails dbm-gorm-diff."  The plugin used to do this like so:
https://github.com/grails-plugins/grails-database-migration/blob/79a2227c99336d11056a11fec97bd4100331f888/src/groovy/grails/plugin/databasemigration/MigrationUtils.groovy#L293
It looks like I could pass a DiffOutputControl to doDiffToChangeLog in https://github.com/yamkazu/database-migration/blob/master/grails-app/commands/org/grails/plugins/databasemigration/command/DbmGormDiffCommand.groovy and that should exclude tables from the diff.  Are you open to PRs for this feature?
Thanks.

@thurisaz Can you please recreate this issue in the main repo (https://github.com/grails-plugins/grails-database-migration)? I have merged this fork back into it. So, would be good to track issues in one spot. Thanks!


Update scrive esign

When submitting a connector, please make sure that you follow the requirements below, otherwise your PR might be rejected. We want to make you have a well-built connector, a smooth certification experience, and your users are happy :)
If this is your first time submitting to GitHub and you need some help, please sign up for this session.

[ ] I attest that the connector doesn't exist on the Power Platform today. I've verified by checking the pull requests in GitHub and by searching for the connector on the platform or in the documentation.
[ ] I attest that the connector works and I verified by deploying and testing all the operations.
[ ] I attest that I have added detailed descriptions for all operations and parameters in the swagger file.
[ ] I attest that I have added response schemas to my actions, unless the response schema is dynamic.
[ ] I validated the swagger file, apiDefinition.swagger.json, by running paconn validate command.
[ ] If this is a certified connector, I confirm that apiProperties.json has a valid brand color and doesn't use an invalid brand color, #007ee5 or #ffffff. If this is an independent publisher connector, I confirm that I am not submitting a connector icon.

If you are an Independent Publisher, you must also attest to the following to ensure a smooth publishing process:

[ ] I have named this PR after the pattern of "Connector Name (Independent Publisher)" ex: HubSpot Marketing (Independent Publisher)
[ ] Within this PR markdown file, I have pasted screenshots that show: 3 unique operations (actions/triggers) working within a Flow. This can be in one flow or part of multiple flows. For each one of those flows, I have pasted in screenshots of the Flow succeeding.
[ ] Within this PR markdown file, I have pasted in a screenshot from the Test operations section within the Custom Connector UI.
[ ] If the connector uses OAuth, I have provided detailed steps on how to create an app in the readme.md.

@microsoft-github-policy-service agree [company="{your company}"]

@microsoft-github-policy-service agree [company="Scrive AB"]

@microsoft-github-policy-service agree company="Scrive AB"

@microsoft-github-policy-service agree company="Scrive AB"

Is this validation error really correct?
I am trying to config multi-auth as per your documentation. Using connectionParameterSets should be ok?

Hello @marcusdelang thank you so much for the fix. Please allow us some time to test this.


Add archive support

Add an iterator over objects in an archive. It would be nice if this same iterator also worked for Mach-O fat binaries, and also returned a single object for files that aren't archives.

goblin already has some support for this too, and using it would avoid another dependency.

Didn't know that.

Fixed in #252. I decided it wasn't useful to try to share anything with Mach-O fat binaries.

#252 added archive read support, does object support write archive?

Not at the moment.


Allow custom JOB_TIME_REQ

Closes #52.

2024-02-18: I am currently testing this on two genomes which previously repeatedly failed for me, because they exceeded the hardcoded JOB_TIME_REQ of two days
2024-02-23: I have confirmed that this pull works for those two repeatedly failing genomes!

I've confirmed that this pull works for those two repeatedly failing genomes in #52! Ready for your review :)

Just bumping!

@kirilenkobm Please excuse my mention! --- just realized my previous bump may have gone into the void, since no one is subscribed to this PR