WIP: DataFrame support

In [5]: import numpy as np
   ...: import pandas as pd
   ...: import zarr
   ...:
   ...: pd.options.display.max_rows=10
   ...: np.random.seed(1234)
   ...: N = 10000000
   ...: ngroups = 1000
   ...: strings = tm.rands_array(10, 1000)
   ...:
   ...: df = pd.DataFrame({'A': np.arange(N),
   ...:                    'B': np.random.randint(0, ngroups, size=N),
   ...:                    'C': np.random.randn(N),
   ...:                    'D': pd.date_range('20130101', periods=ngroups).take(np.random.randint(0, ngroups, size=N)),
   ...:                    #'E': pd.Series(strings.take(np.random.randint(0, ngroups, size=N))).astype('category')})
   ...:                    #'f': strings.take(np.random.randint(0, ngroups, size=N)),
   ...:                    })
   ...: g = zarr.group('foo')
   ...:
   ...:

In [6]: g['df'] = df

In [7]: g['df']
Out[7]:
Frame(/df, (10000000, 4), chunks=(19532, 1), )
 store: DirectoryStore

 Array(/df/data/A, int64, order=C)
  nbytes: 76.3M; nbytes_stored: 1.2M; ratio: 65.8; initialized: 512/512
  compressor: Blosc(cname=u'lz4', clevel=5, shuffle=1)
 Array(/df/data/B, int64, order=C)
  nbytes: 76.3M; nbytes_stored: 17.3M; ratio: 4.4; initialized: 512/512
  compressor: Blosc(cname=u'lz4', clevel=5, shuffle=1)
 Array(/df/data/C, float64, order=C)
  nbytes: 76.3M; nbytes_stored: 72.5M; ratio: 1.1; initialized: 512/512
  compressor: Blosc(cname=u'lz4', clevel=5, shuffle=1)
 Array(/df/data/D, int64, order=C)
  nbytes: 76.3M; nbytes_stored: 51.9M; ratio: 1.5; initialized: 512/512
  compressor: Blosc(cname=u'lz4', clevel=5, shuffle=1)
In [9]: g['df'][:]
Out[9]:
               A    B         C          D
0              0  324  1.087268 2014-10-03
1              1  918 -0.094611 2013-08-25
2              2  800 -0.192653 2014-05-02
3              3  335  1.888183 2014-02-12
4              4  227 -0.741727 2013-01-24
...          ...  ...       ...        ...
9999995  9999995  103  0.213191 2014-05-03
9999996  9999996  823  0.548665 2013-11-24
9999997  9999997  166  1.382537 2014-10-13
9999998  9999998  606  0.368681 2014-08-15
9999999  9999999  873  0.183906 2015-08-23

[10000000 rows x 4 columns]

very preliminary WIP for DataFrame support directly in zarr. This adds a new meta type like Array, Group, called Frame. It has top-level metadata (nrows, columns, dtypes) and stores the data in individual Arrays. These can be individuall compressed via different compressors, if the default is specified. (I would propose the default be what @mrocklin mentioned in castra to start).
Uses @alimanfoo pickle filter directly. chunking is done (but no selection yet). Currently just supporting set/getitem access for columnar access.
I would propose adding:
g['df'].loc[row_indexer, columns]

for setting / getting (main for chunking support). This preserves the pandas semantics of [] for columnar access, while .loc is for both row/columns. This deviates a bit from zarr semantics though.
minimal testing ATM, should be able to support all numpy dtypes that pandas supports; categorical soon. I had to add the notion of a storage dtype and a semantic one (eg. M8[ns] is actually stored as i8, so the actual store shows that above). This is an implementation detail but I don't think a big deal. Need this for example, for category support (where we actually have a sub-group with codes/categories).
I was trying to isolate pandas so that it would not be a requirement unless you actually tried to save a DataFrame. Currently it must be there to import zarr, but I think that could be relaxed.
I extracted some things from Array and created a Base, more could be done here.
Finally I think it might be worth adding a filter which does arrow encoding for object arrays (in reality just a pair of cython functions could do it without making feather a dep, e.g. https://github.com/wesm/feather/blob/master/python/feather/interop.h) and then saving as values, offsets (like how we do for categories). Might be interesting / better performed than pickle. IDK.
xref https://github.com/dask/dask/issues/1599

cc @alimanfoo
cc @mrocklin
cc @wesm
cc @shoyer

Cool to see work in this direction.  Two questions:
Why the need for a separate Frame type?  I wonder if a dataframe could just be a Group that is laid out in a certain way.
Is this in scope for Zarr?  I wonder if something like this should be external.  It would be nice to see Zarr stay minimal and somewhat general.

Frame is essentially a Group that is laid out. But a couple of issues that make it special.

you want to hide the actual storage format (e.g. say I want to store dates as i8), but re-serialize to M8[ns]. This could in theory actually be handled by Array
I may want to specify how to chunk overall, and potentially filters on a Dataframe basis and have it consistent across all columns.

Sure this could be a separate package, zframe or whatever. Though re-uses lots of the zarr machinery (which does multiple things, e.g. storage backends, hierarchies, and arrays).

updated the top. forgot the object column :<

Perhaps things like logical vs physical dtype mappings for dtypes could be stored in the group's metadata?
Your point about filtering across all columns does serve as pretty good motivation for a separate class.  Some special API on top of groups would be definitely desired.  Perhaps the Frame class could encapsulate a Zarr Group rather than re-implement its logic?
I think in my mind Pandas is mired in the unfortunate mess that is the real world while Zarr is still fairly clean.  I'm somewhat in favor of building messy things on top of clean things, rather than within them.  However the corollary here is that clean things need to make sure that people can extend them externally with ease.  Perhaps there are lessons to be learned from this work?

Any thoughts from others on this topic?

Just a note -- it looks like you don't make use of date/time types directly, but I realized that these are ambiguously describe in the zarr spec (#85).
I'm also not sure this makes sense for zarr proper. Adding a query language to zarr (.loc) feels like a significant increase in scope, as does a specific serialization strategy for pandas's logical dtypes.

@jreback indeed very cool to see work in this direction. Here’s a few thoughts, very open to discussion on any of these points.
I have a strong desire to keep zarr lean and mean. Partly that’s because I am cautious about taking on any more code commitments, but also because I want to make sure zarr fills a well-defined niche and complements other libraries as much as possible. In particular, the original motivation for zarr was to provide a storage partner to dask.array that made the most of the parallel computing capabilities of dask. That helped to give a clear scope, i.e., it was clear what problem I was trying to solve, which was how to design a chunked compressed N-dimensional array store that enabled writing in parallel to different regions of an array. There were also some simplifying assumptions I could make which really helped to reduce complexity. Much of the original inspiration for zarr came from the carray class in bcolz, but because I was targeting a dask.array usage context, I could do away with some things, e.g. carray has several optimisations for iterating over items and for appending to an array one item at-a-time, which I did not need to include in zarr.
I say this because the interesting question that @mrocklin has now raised is, what would an ideal storage partner for dask.dataframe look like? I know a lot less about dask.dataframe and about out-of-core processing of tabular data in general, and so I have much less of a feel for what the problem we’re trying to solve is. I think what I would like to achieve in the short term is some set of minimal functionalities or enhancements or example code, either within zarr or pandas or dask or some sandbox package, that would allow @mrocklin to do some experimentation with dask.dataframe and zarr, and get some insight on where there could be significant benefits and synergy. I.e., where the niche is; what are the key problems not solved or at least not solved well by existing storage libraries? I am hoping @mrocklin and others can advise on what further minimal work would be required enable this experimentation. I’d like to proceed slowly because as I say I don’t understand the use cases and algorithms very well, so I don’t have a good sense of where there could be a lot of bang for buck, and anyway I’m sure some experiments are required and there may be some surprises.
Having said all that, I am open to the idea that zarr might be extended to include some kind of Frame or Table class. I’d be happy to explore this, but I think there are a number of questions that need careful discussion.
One question is, what would be the public API for a zarr Frame class? I.e., what are the public methods, including features supported via getitem and setitem? Clearly the natural thing to do is stick as close as possible to pandas, but where does zarr stop and dask.dataframe take over? To give an analogy, zarr.Array only supports indexing with slices; fancy indexing is left for dask.array or some other higher-level library. So what are the minimal set of features for Frame that would fill an important niche?
Another question is, what type(s) of parallel processing would be anticipated zarr.Frame being used for, and how would zarr.Frame implement synchronization/locking within the public API to provide maximum support? To give another anology, zarr.Array was designed for use as the data destination for chunked algorithms, and so allows writing of contiguous regions via setitem, with write locking done at the chunk level so different regions of the array can be written to in parallel. What are the major parallel use cases and synchronisation issues for a Frame class?
A third question is, technically, how to implement a Frame class, and what (if any) changes to the storage spec are required? In this PR you created a new class sibling to Array and Group. An alternative suggested by @mrocklin would be to sub-class or wrap the Group class. I tend to favour sub-classing Group as it could imply little or no changes to the storage spec, layering the functionality on the existing Group and metadata infrastructure. But there may be good architectural and/or performance/scalability reasons for doing otherwise that I am not aware of, so I don’t want to advocate strongly for any particular direction at this time and would rather delay making any commitment until knowing more.
A final question is, what would zarr.Frame do that bcolz.ctable doesn’t do? Are they attempting to fill the same niche, or not? If not, what are the differences in target use cases? If so, then why is it better to implement in Zarr rather than work from the ctable class? Again I don’t want to stifle any discussion, but I think I’d like to understand more about what problem(s) we are trying to provide a better solution for, and then much of this would become clear. In any case it would be good to include @FrancescAlted in the conversation.
Sorry for the long-winded response, again very cool to see this work, and very interested to hear thoughts.

@alimanfoo thanks for your response. So here's some philosophy and rationale that hopefully answers some questions (and brings up some more!). this is probably TL; DR. but DO read it!
I have recently been using dask to work with fairly large remotely stored data. This is a mix of dtypes, including some large text fields (with varying distribution of length within the field), numerics (mostly floats) and dates. This is hosted on a local S3 cluster (so network access is quite cheap, compared to a typical S3 on aws installation). The data is coming to me initially in .txt.gz (csv gzipped).
To put some numbers on this. I have about 2000 files, which when expanded are 1.5B rows, say 20 columns (several TB's in total). However at any one time I only really load up a couple of columns into distributed memory (currently using about 1.5TB memory across a few machines with 150 cores ATM).
This works pretty well now. IOW I can load it up, do some analysis, then .restart() the cluster and do some more. However this got me thinking about some storage options in order to improve the experience.
In an ideal scenario, I would perform a pre-processing step where I read in the data, clean (including dtype conversions), maybe re-partition, and re-store to s3 in a binary format.
Some conversions are pretty cheap so doing them after loading up from storage are cheap, so post-processing is not a big deal. IOW things like pd.to_datetime across partitions. Ideally I would also like to .categorize across the entire set (per-column of course), and re-store, but this can be fairly expensive to do.
Bottom line is any 'real' post-load-transforms requires a storage format to:

store dtypes (metadata)
be s3 friendly
be columns accessible
deal with object dtype nicely
dataframe friendly (meaning how much work to support various dtypes)

So I did a mini-test of some storage formats. Here are my thoughts (no particular order), and obviously repeating some known things.

gzipped csv

pros: familiar, s3 friendly (using s3fs)
cons: no per-column access, no meta data storage, slowish (but usually pretty adequate), s3 friendly, though gzip is detrimental to this.

HDF5 with various compression options (lzma, blosc)

pros: (keeps meta data), some per-column access, dataframe friendly
cons: not friendly to object dtypes as has fixed string storage (compression does help with though though)

gzipped feather (I also tried lzma but that was strictly worse)

pros: per-column access (though still must unzip the entire file, so this may be a con), efficient storing of objects (via automatic categorization & layout behind the scenes). dataframe friendly.

not compressed or chunked internally (not that much of a con in my case as I have a natural partition already)

castra

pros: compressed, chunked, dataframe friendly
cons: not supported any more :<, IIRC categoricals are handled, but not efficient for object dtypes

bcolz

pros: per-column access, chunked, compressed
cons:  defaults blow up dataframe object columns to full storage (negating the compression). this can be fixed by categorizing, but requires a bit of user effort; also not easily able to store the categorical WITH the original frame (IOW, the bcolz groups don't allow an easy way to mix in OTHER objects w/o them being attrs). biggest con is that this generates lots of files, so is not too friendly to s3.

zarr

pros: per-column access, chunked, compressed

cons: not dataframe friendly, issues with object dtypes

So every format has various tradeoffs. Note that I am not being critical of any author here, rather the reverse. I am looking at a usecase which I think is actually pretty common, but not fully addressed.
So feather is quite friendly, but doesn't have the per-column access (and so must unzip the entire file), while bcolz would work, gets a little awkward once you need to move it to an external file system like s3.
So I liked the base that zarr provided. It seems building a zframe format on top is the right thing to do. It has 2 really nice properties. a HDF5 like interface which is familiar, and it has a ZipStore, which makes it really s3 friendly.
I am indifferent where to put this:

inside zarr: natural for users, does make more complex though
zframe: seems straightforward and could provide a nice interface.

Thanks @jreback, very interesting. FWIW I think working on this in a
separate package at least for now would be the best option, would at least
give you more freedom to experiment. My only suggestion would be to try and
implement Frame by sub-classing or wrapping zarr.Group if possible and
layering on top of existing Zarr API and storage spec, i.e., using the
custom attributes on Array and Group objects etc. If there are good reasons
to deviate this then of course fine.
On Wed, Oct 19, 2016 at 12:08 PM, Jeff Reback<EMAIL_ADDRESS>wrote:

@alimanfoo https://github.com/alimanfoo thanks for your response. So
here's some philosophy and rationale that hopefully answers some questions
(and brings up some more!). this is probably TL; DR. but DO read it!
I have recently been using dask to work with fairly large remotely stored
data. This is a mix of dtypes, including some large text fields (with
varying distribution of length within the field), numerics (mostly floats)
and dates. This is hosted on a local S3 cluster (so network access is quite
cheap, compared to a typical S3 on aws installation). The data is coming to
me initially in .txt.gz (csv gzipped).
To put some numbers on this. I have about 2000 files, which when expanded
are 1.5B rows, say 20 columns (several TB's in total). However at any one
time I only really load up a couple of columns into distributed memory
(currently using about 1.5TB memory across a few machines with 150 cores
ATM).
This works pretty well now. IOW I can load it up, do some analysis, then
.restart() the cluster and do some more. However this got me thinking
about some storage options in order to improve the experience.
In an ideal scenario, I would perform a pre-processing step where I read
in the data, clean (including dtype conversions), maybe re-partition,
and re-store to s3 in a binary format.
Some conversions are pretty cheap so doing them after loading up from
storage are cheap, so post-processing is not a big deal. IOW things like
pd.to_datetime across partitions. Ideally I would also like to .categorize
across the entire set (per-column of course), and re-store, but this can be
fairly expensive to do.
Bottom line is any 'real' post-load-transforms requires a storage format
to:

store dtypes (metadata)
be s3 friendly
be columns accessible
deal with object dtype nicely
dataframe friendly (meaning how much work to support various dtypes)

So I did a mini-test of some storage formats. Here are my thoughts (no
particular order), and obviously repeating some known things.

gzipped csv

pros: familiar, s3 friendly (using s3fs)

cons: no per-column access, no meta data storage, slowish (but
usually pretty adequate), s3 friendly, though gzip is detrimental
to this.

HDF5 with various compression options (lzma, blosc)

pros: (keeps meta data), some per-column access, dataframe friendly

cons: not friendly to object dtypes as has fixed string storage
(compression does help with though though)

gzipped feather (I also tried lzma but that was strictly worse)

pros: per-column access (though still must unzip the entire file, so
this may be a con), efficient storing of objects (via automatic
categorization & layout behind the scenes). dataframe friendly.

not compressed or chunked internally (not that much of a con in
my case as I have a natural partition already)

castra

pros: compressed, chunked, dataframe friendly

cons: not supported any more :<, IIRC categoricals are handled,
but not efficient for object dtypes

bcolz

pros: per-column access, chunked, compressed

cons: defaults blow up dataframe object columns to full storage
(negating the compression). this can be fixed by categorizing, but requires
a bit of user effort; also not easily able to store the categorical WITH
the original frame (IOW, the bcolz groups don't allow an easy way to mix in
OTHER objects w/o them being attrs). biggest con is that this generates
lots of files, so is not too friendly to s3.

zarr

pros: per-column access, chunked, compressed

cons: not dataframe friendly, issues with object dtypes

So every format has various tradeoffs. Note that I am not being critical
of any author here, rather the reverse. I am looking at a usecase which I
think is actually pretty common, but not fully addressed.
So feather is quite friendly, but doesn't have the per-column access (and
so must unzip the entire file), while bcolz would work, gets a little
awkward once you need to move it to an external file system like s3.
So I liked the base that zarr provided. It seems building a zframe format
on top is the right thing to do. It has 2 really nice properties. a HDF5
like interface which is familiar, and it has a ZipStore, which makes it
really s3 friendly.
I am indifferent where to put this:

inside zarr: natural for users, does make more complex though
zframe: seems straightforward and could provide a nice interface.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/pull/84#issuecomment-254782455, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8Qgyaqs2ZLCVCrGMZ8StibGwjizUkks5q1fo1gaJpZM4KX1em
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email<EMAIL_ADDRESS>Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

going to close this for now.
parquet support is now fast & first class: https://github.com/martindurant/fastparquet/ (very recent), thanks @martindurant
I know @wesm is integrating this into pyarrow as well.
So in theory this is a good idea (prob as a separate package, maybe zframe), this would require substantial investments in time / effort for benchmarking.
I do think that this is a nice alternative format.

Thanks @jreback, happy to revisit in future if there is interest.


Concatenate sweep files

In response to https://github.com/apcooper/desi_mws_projects/issues/13. Do this in sweeps.io?

Fixed in 6f4604f18b211701b536c0b6489f496445b73ed2.


Error handling during graph evaluation

Currently error handling is close to non-existent: We panic when something goes wrong during graph evaluation (i.e. a parameter we expect to be present is disconnected). This was (somewhat) fine for use as a library but it severely falls short when working on the upcoming graphical editor.
If there is an error on the graph, we want the editor to report the error and let the user resolve it, not to crash.
We can probably introduce some graph evaluation error type (e.g. GraphEvalError), and change return types of the NodeLike methods that can fail to Result<T, GraphEvalError> for the relevant type T. The errors can then propagate up the graph and the editor will report it.

Closed by #26


Can not build project with npm install & npm start

Following the readme file in the example folder, I want to build the project with npm install & npm start command. However it pop up the error message like "Can not resolve knockout "

Whoops, knockout is missing from that package.json. npm install knockout should fix, and I'll add it to package.json.

Yes, actually I found that and fix it. :) Thank you anyway
On Tue, Apr 4, 2017 at 2:33 AM, Casey Webb<EMAIL_ADDRESS>wrote:

Whoops, knockout is missing from that package.json. npm install knockout
should fix, and I'll add it to package.json.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/Profiscience/ko-component-router/issues/181#issuecomment-291161460,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGwe5mBnu8jclSOQbfIU3e6SRlv5DaTlks5rsQMmgaJpZM4MxBIe
.


Certain Interfaces causes big fps drops

Whilst Minecraft does natively lower fps in container screens to save resources, the latest version of this mod significantly decreases the frames on certain screens. The two in which I've noticed the irregularity is are the crafting interface screen and the mod screen. I have attached pictures of the fps in certain screens below:
Normal game fps:

Inventory fps(Normal decrease):

Crafting Interface decrease(Irregular decrease):

Classic Crafting Interface(Normal Decrease):

Creative Inventory Screen(Normal Decrease)::

Mod Screen(Irregular Decrease):

I'm not so sure what about the Crafting Interface screen causes a drop, since the Creative Screen which draws many more images is lowered in a reasonable manner, but this has only started as of recently with the latest version.

Yes, I'm aware of this, and most of this performance drop is due to the game's poorly optimized rendering. To be more specific, the Crafting Interface always checks if you can craft the items on the screen, which causes a certain loss of performance, and has translucent items most of the time, along with the mods screen that has the tooltip box which is also translucent, and this translucency is mainly responsible for this drop in FPS, and there is not much point in changing the texture, due to the fact that it is enabled in the code


Unable to find environment-variables

https://owncloud.github.io/ocis/configuration/#environment-variables :
If you prefer to configure the service with environment variables you can see the available variables below.
Nope, I can't see them. Where would I find this list?

The env vars are noted after the cli arg separated by the pipe.
-log-pretty | $OCIS_LOG_PRETTY
Enable pretty logging. Default: false.

In the example above OCIS_LOG_PRETTY is the env-var.

Huh? I have no clue what you mean. I'm asking where the [missing] list/table of env vars is located.

You can find the env vars in the respective extension configuration section eg. https://owncloud.github.io/extensions/proxy/configuration/

Are you trolling me? I report that that link is missing the entire env var table, and you link me back to it?

I'm just going to close this out. I'll stick to Nextcloud. I wish you guys the best, you need it.

Are you trolling me? I report that that link is missing the entire env var table, and you link me back to it?

We are not trolling you. The env var tables are spread across the different extensions. Therefore you only have a few env vars on the main command / your given link. As my colleague tried to explain that the env vars are one section below your linked section:

The other env vars are available on the respective extension configuration page: lhttps://owncloud.github.io/extensions/extension-name/configuration/ eg. https://owncloud.github.io/extensions/proxy/configuration/

I'm not looking to output logs. What would posses you pipe logs into [the location of] some variable? Is this some Windows convention? I'm perfectly OK with letting docker/kubernetes handle the logs. I would like to use the stock entry point.

That's not a bash fragment other than the --option, is it? Can you guys please change the docs to replace the pipe (|) with an or (||) or better yet, use a syntax like:
CLI: --option   ENV: $ENV_VAR


[BUG] File Manager not working on 0.4.11

Module Version: 0.4.11
Describe the bug
The File Manager will no longer launch. Rolling back to 0.4.10 resolves the issue. I have tried on local foundry as well as the forge in new and existing worlds.
To Reproduce
Click File Manager button or view error in console on world launch.
Screenshots

Browser:
Chrome or native foundry app
Foundry Version:
v10 288 and 290
Game System:
dnd5e

I also see this error when trying to manually rebuild the index on v.4.11

Additionally, the index seems to get reset on every page refresh, but the index.js file doesn't get erased from the file system, so it's there just not being read? Which may trigger the first error?

I made a "hot"fix: https://github.com/averrin/alpha-suit/releases/tag/0.4.12
I hope it helps. If not, feel free to reopen this issue.


Remove 'plugin not found' warning when inheriting workspace dependencies

Problem
Main scarb.toml:
[workspace.dependencies]
cairo_test = "2.8.2"

Workspace's member scarb.toml:
[dev-dependencies]
cairo_test.workspace = true

Result in:
warn: `cairo_test` plugin not found
please add the following snippet to your Scarb.toml manifest:
\```
[dev-dependencies]
cairo_test = "2.8.2"
\```

Steps
No response
Possible Solution(s)
No response
Notes
No response
Version
scarb 2.8.2 (a37b4cbfc 2024-09-09)
cairo: 2.8.2 (https://crates.io/crates/cairo-lang-compiler/2.8.2)
sierra: 1.6.0

Hi @julio4 !
Can you provide a reproduction for this?
I failed to reproduce in https://github.com/software-mansion/scarb/tree/main/examples/workspaces for 2.8.2

You can reproduce with:
checkout https://github.com/NethermindEth/StarknetByExample/pull/246
cd listings/getting-started/storage
scarb cairo-test


UAT Round 2 | E-filing | Organization Drop down

As per the User stories, if a case is filed against an organization, the organization field should perform a fuzzy seach and show a drop down of matching organizations.
Expected behavior

Suppose a case is filed against organization Mathews and Sons
Now, when any user tries to file a new case, in the org drop down, if M is typed and further Ma is typed and so on, a list of matching organizations appears in the dropdown.

@SureshSoren @Ramu-kandimalla this is mentioned in scope and not an enhancement
cc @subhashini-egov


NULL to EMPTY_STRING

when i setup connection option as  PDO::ATTR_ORACLE_NULLS =>PDO::NULL_EMPTY_STRING
and run
$db->insert("table", [ 	"name" => "foo", 	"email" =><EMAIL_ADDRESS>	"phone" => "" ]);
the phone field still insert as empty string not null , WHY??

If you want to insert null data, try this:
$db->insert("table", [
    "name" => "foo",
    "email" =><EMAIL_ADDRESS>    "phone" => $phone === "" ? null : $phone
]);

so the PDO::NULL_EMPTY_STRING  doesn't work??

Because the empty string "" is still mapped as PDO::PARAM_STR inside https://github.com/catfan/Medoo/blob/master/src/Medoo.php#L744
Of course, you can just create your insert() method to handle those values and then call Medoo's insert().
function myInsert($data) {

    $result = [];

    foreach ($data as $key => $value) {
        $result[$key] = $value === "" ? null : $value
    }

    $database->insert("table", $result);
}


chore: contribution generation

#8  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @lewxdev and the rest of your teammates on  Graphite

migrated most changes to a more manageable stacked PR


refactor(interval): ♻ rename fromQualitySemitones factory constructor

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1035

Relevant Lines:
1035

💛  - Coveralls


UE4Game have to be rebuilt when packaging game from editor (windows)

Make a full build in Visual Studio 2013, notice XGConsole been called, build success without error
Start a project in UE4 editor, try to package the game, UBT will try to rebuild UE4Game even it's already built in Visual Studio, also noticed XGConsole not been called from console log

What version UE4 are you using?

UBT will try to rebuild UE4Game even it's already built in Visual Studio

May be reated: https://github.com/EpicGames/UnrealEngine/pull/1678

Also it would be nice to look at the log of the build without the use xgConsole.

4.10. Just tried your PR, it's still the same rebuilding UE4Game behavior when packaging.
here's part of the log:
MainFrameActions: Packaging (Windows (64-bit)): Running AutomationTool...
MainFrameActions: Packaging (Windows (64-bit)): path="...[path here]"
MainFrameActions: Packaging (Windows (64-bit)): Program.Main: Executed from AutomationToolLauncher (C:\Users\cc\AppData\Local\assembly\dl3\HXDB91HJ.8ZG\55VK0AAP.G6Q\15cc220c\d7d903f8_782bd101\AutomationTool.exe)
MainFrameActions: Packaging (Windows (64-bit)): Program.Main: CWD=E:\dev\unreal\src\410x\Engine\Binaries\DotNET
MainFrameActions: Packaging (Windows (64-bit)): Automation.ParseCommandLine: Parsing command line: BuildCookRun -nocompileeditor -nop4 -project=E:/projects/unreal/TestOnly/TestCube2/TestCube2.uproject -cook -stage -archive -archivedirectory=E:/projects/unreal/_Export/TestWinLiveReload -package -clientconfig=Development -ue4exe=UE4Editor-Cmd.exe -CookAll -pak -prereqs -nodebuginfo -targetplatform=Win64 -build -CrashReporter -utf8output -compile
MainFrameActions: Packaging (Windows (64-bit)): Automation.Process: IsBuildMachine=False
MainFrameActions: Packaging (Windows (64-bit)): Automation.Process: ShouldKillProcesses=True
MainFrameActions: Packaging (Windows (64-bit)): Automation.Process: Setting up command environment.
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.SetEnvVar: SetEnvVar uebp_EngineSavedFolder=E:/dev/unreal/src/410x/Engine/Programs/AutomationTool/Saved
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.DeleteDirectoryContents: DeleteDirectoryContents(E:/dev/unreal/src/410x/Engine/Programs/AutomationTool/Saved/Logs)
MainFrameActions: Packaging (Windows (64-bit)): ProjectUtils.CleanupFolders: Cleaning up project rules folder
MainFrameActions: Packaging (Windows (64-bit)): Automation.Process: Compiling scripts.
MainFrameActions: Packaging (Windows (64-bit)): ScriptCompiler.CleanupScriptsAssemblies: Cleaning up script DLL folder
MainFrameActions: Packaging (Windows (64-bit)): ScriptCompiler.FindAndCompileScriptModules: Building Automation projects in parallel...
MainFrameActions: Packaging (Windows (64-bit)): MSBuild: Microsoft(R) ��������汾 <IP_ADDRESS>
MainFrameActions: Packaging (Windows (64-bit)): MSBuild: [Microsoft .NET Framework �汾 4.0.30319.42000]
MainFrameActions: Packaging (Windows (64-bit)): MSBuild: ��Ȩ���� (C) Microsoft Corporation����������Ȩ����
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   DotNETUtilities -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\DotNETUtilities.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   EnvVarsToXML -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\EnvVarsToXML.exe
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   UnrealBuildTool -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\UnrealBuildTool.exe
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   AutomationUtils.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationUtils.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   AllDesktop.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\AllDesktop\AllDesktop.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   GUBP.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\GUBP.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   AutomationScripts.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\AutomationScripts.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   Android.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\Android\Android.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   HTML5.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\HTML5\HTML5.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   DeploymentInterface -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\IOS\DeploymentInterface.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   IOS.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\IOS\IOS.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   Linux.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\Linux\Linux.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   Mac.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\Mac\Mac.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   Win.Automation -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\AutomationScripts\Win.Automation.dll
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: Took 2.4407258s to run MSBuild.exe, ExitCode=0
MainFrameActions: Packaging (Windows (64-bit)): ScriptCompiler.LoadPreCompiledScriptAssemblies: Loading precompiled script DLLs
MainFrameActions: Packaging (Windows (64-bit)): ScriptCompiler.LoadPreCompiledScriptAssemblies: Found 9 script DLL(s).

MainFrameActions: Packaging (Windows (64-bit)): MSBuild: _CopyAppConfigFile:
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   ���ڽ��ļ��ӡ�app.config�����Ƶ���..\..\..\Binaries\DotNET\UnrealBuildTool.exe.config����
MainFrameActions: Packaging (Windows (64-bit)): MSBuild: CopyFilesToOutputDirectory:
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   ���ڽ��ļ��ӡ�obj\Development\UnrealBuildTool.exe�����Ƶ���..\..\..\Binaries\DotNET\UnrealBuildTool.exe����
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   UnrealBuildTool -> E:\dev\unreal\src\410x\Engine\Binaries\DotNET\UnrealBuildTool.exe
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:   ���ڽ��ļ��ӡ�obj\Development\UnrealBuildTool.pdb�����Ƶ���..\..\..\Binaries\DotNET\UnrealBuildTool.pdb����
MainFrameActions: Packaging (Windows (64-bit)): MSBuild: �����������Ŀ��E:\dev\unreal\src\410x\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj��(Rebuild ��Ŀ��)�Ĳ�����
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:
MainFrameActions: Packaging (Windows (64-bit)): MSBuild: �ѳɹ����ɡ�
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:     0 ������
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:     0 ������
MainFrameActions: Packaging (Windows (64-bit)): MSBuild:
MainFrameActions: Packaging (Windows (64-bit)): MSBuild: ����ʱ�� 00:00:02.46
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: Took 2.7889723s to run MSBuild.exe, ExitCode=0
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: E:\dev\unreal\src\410x\Engine\Binaries\DotNET\UnrealBuildTool.exe UE4Game Win64 Development   -remoteini="E:\projects\unreal\TestOnly\TestCube2" -noxge -generatemanifest -NoHotReloadFromIDE
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: Took 3.9447891s to run UnrealBuildTool.exe, ExitCode=0
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.PrepareManifest: Copied UBT manifest to E:\dev\unreal\src\410x\Engine\Programs\AutomationTool\Saved\Logs\UBTManifest.0.xml
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: E:\dev\unreal\src\410x\Engine\Binaries\DotNET\UnrealBuildTool.exe UE4Game Win64 Development   -remoteini="E:\projects\unreal\TestOnly\TestCube2" -noxge -NoHotReloadFromIDE -ignorejunk
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: Creating makefile for UE4Game (UnrealBuildTool.exe is newer)
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: Performing 262 actions (8 in parallel)
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: PCH.SlateBasics.h.cpp
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: Module.LightPropagationVolumeRuntime.cpp
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: PCH.EnginePrivate.h.cpp

Looks like problem in Engine/Source/Programs/AutomationTool/AutomationUtils/UE4Build.cs
Function BuildWithUBT in lines 1354 and 1416 is runned with DisableXGE = true. Moreover, I did not find it DisableXGE = false.

According to log:
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.Build: XGE was requested, but is unavailable, so we won't use it.

Unreal Automation Tool can't find xgConsole.exe in PATH environment variable.
This problem was solved by https://github.com/EpicGames/UnrealEngine/pull/986 and merged to 4.10

It seems I have to restart computer after installation to make PATH environment variable kick in.
Now it still rebuilding UE4Game when packaging, but this time it's building with XGConsole.
Here's the log:
MainFrameActions: Packaging (Windows (64-bit)): Program.Main: Executed from AutomationToolLauncher
...
MainFrameActions: Packaging (Windows (64-bit)): Automation.Process: IsBuildMachine=False
MainFrameActions: Packaging (Windows (64-bit)): Automation.Process: ShouldKillProcesses=True
MainFrameActions: Packaging (Windows (64-bit)): Automation.Process: Setting up command environment.
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.SetEnvVar: SetEnvVar uebp_EngineSavedFolder=E:/dev/unreal/src/410x/Engine/Programs/AutomationTool/Saved
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.DeleteDirectoryContents: DeleteDirectoryContents(E:/dev/unreal/src/410x/Engine/Programs/AutomationTool/Saved/Logs)
MainFrameActions: Packaging (Windows (64-bit)): ProjectUtils.CleanupFolders: Cleaning up project rules folder
MainFrameActions: Packaging (Windows (64-bit)): Automation.Process: Compiling scripts.
MainFrameActions: Packaging (Windows (64-bit)): ScriptCompiler.CleanupScriptsAssemblies: Cleaning up script DLL folder
MainFrameActions: Packaging (Windows (64-bit)): ScriptCompiler.FindAndCompileScriptModules: Building Automation projects in parallel...
...
MainFrameActions: Packaging (Windows (64-bit)): ProjectParams.ValidateAndLog: Project Params **************
MainFrameActions: Packaging (Windows (64-bit)): Project.Build: ********** BUILD COMMAND STARTED **********
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.Build: ************************* UE4Build:
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.Build: ************************* ForceMonolithic: False
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.Build: ************************* ForceNonUnity:False
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.Build: ************************* ForceDebugInfo: False
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.Build: ************************* UseXGE: True
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.Build: ************************* UseParallelExecutor: False
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe E:/dev/unreal/src/410x/Engine/Source/Programs/UnrealBuildTool/UnrealBuildTool.csproj /verbosity:normal /target:Rebuild /property:Configuration=Development /property:Platform=AnyCPU
...
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: Took 1.7532355s to run MSBuild.exe, ExitCode=0
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: E:\dev\unreal\src\410x\Engine\Binaries\DotNET\UnrealBuildTool.exe UE4Game Win64 Development -generatemanifest -nobuilduht -xgeexport  -remoteini="E:\projects\unreal\TestOnly\TestCube2" -NoHotReloadFromIDE
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: Performing full C++ include scan (building a new target)
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: XGEEXPORT: Exported '../../Engine/Intermediate/Build/UBTExport.0.xge.xml'
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: XGE execution time: 8.90 seconds
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: Took 8.9303167s to run UnrealBuildTool.exe, ExitCode=0
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.PrepareManifest: Copied UBT manifest to E:\dev\unreal\src\410x\Engine\Programs\AutomationTool\Saved\Logs\UBTManifest.0.xml
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: E:\dev\unreal\src\410x\Engine\Binaries\DotNET\UnrealBuildTool.exe CrashReportClient Win64 Development -generatemanifest -nobuilduht -xgeexport  -NoHotReloadFromIDE -ignorejunk
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: Performing full C++ include scan (building a new target)
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: Parsing headers for CrashReportClient
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool:   Running UnrealHeaderTool CrashReportClient "E:\dev\unreal\src\410x\Engine\Intermediate\Build\Win64\CrashReportClient\Development\UnrealHeaderTool.manifest" -LogCmds="loginit warning, logexit warning, logdatabase error"
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: Reflection code generated for CrashReportClient in 0.8157332 seconds
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: XGEEXPORT: Exported '../../Engine/Intermediate/Build/UBTExport.0.xge.xml'
MainFrameActions: Packaging (Windows (64-bit)): UnrealBuildTool: XGE execution time: 3.31 seconds
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: Took 3.3343452s to run UnrealBuildTool.exe, ExitCode=0
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.PrepareManifest: Copied UBT manifest to E:\dev\unreal\src\410x\Engine\Programs\AutomationTool\Saved\Logs\UBTManifest.1.xml
MainFrameActions: Packaging (Windows (64-bit)): UE4Build.ProcessXGEItems: Running C:\Program Files\Octobuild\xgConsole.exe *******
MainFrameActions: Packaging (Windows (64-bit)): CommandUtils.Run: Run: C:\Program Files\Octobuild\xgConsole.exe "E:\dev\unreal\src\410x\Engine\Programs\AutomationTool\Saved\Logs\UAT_XGE.xml" /Rebuild /MaxCPUS=200
MainFrameActions: Packaging (Windows (64-bit)): xgConsole: xgConsole (0.1.3-x86_64-windows, rev: 71af3c688, rustc: 1.4.0):
MainFrameActions: Packaging (Windows (64-bit)): xgConsole:   C:\Program Files\Octobuild\xgConsole.exe
MainFrameActions: Packaging (Windows (64-bit)): xgConsole:   E:\dev\unreal\src\410x\Engine\Programs\AutomationTool\Saved\Logs\UAT_XGE.xml
MainFrameActions: Packaging (Windows (64-bit)): xgConsole:   /Rebuild
MainFrameActions: Packaging (Windows (64-bit)): xgConsole:   /MaxCPUS=200
MainFrameActions: Packaging (Windows (64-bit)): xgConsole: #5 1/264: Module.LightPropagationVolumeRuntime.cpp

I tried uninstalled octobuild, rebuild engine, packaging in editor is fine then.

[Update]
If I rebuilt whole engine and editor with Visual Studio 2015, then packaging in editor with octobuild installed works normally.

Also you will have troubles with UnrealHeaderTool compilation without https://github.com/EpicGames/UnrealEngine/pull/1825

Related links "Setting environment variables requires reboot on 64-bit":

http://stackoverflow.com/questions/6979741/setting-environment-variables-requires-reboot-on-64-bit
http://stackoverflow.com/questions/22094727/must-reboot-after-changing-path
https://msdn.microsoft.com/en-us/library/aa372883(v=vs.85).aspx

Thanks bozaro!
I'm stick with visual studio 2015 for now. It seems works very well with octobuild.


Is there a way to solve the issue of importing a large amount of data?

I tried importing around 400 nodes of data, but the web browser froze completely. Is there any way to help with the data import process?

you will find its about 180-250 nodes is about reasonable. I rewrote some code to speed up node rendering on the drawflow library, some other posts here from Jero make useful suggestions.  What i would say is keep your node limit <250/200. Too much on the screen anyway makes the experience impossible.
The real trick is to use the modules.
When loading the JSON

Create tabs, set the attribute and click and labels map to tabs
Load in about 150-250 items per tab max
Split up any flows into logical tabs and link there with a flow in and out or connector
On click of tab clear the drawflow area, load the module data for that tab
One thing to think about is making sure all your nodes on current tab keep state (any UI HTML changes) if you have them visible in the node itself (using html save api in the library on node losing focus before switching tabs

It's a bit of work but not too onerous with reasonable web script skills, and you can build 50, 100, 150 tabs and the switching loading of data can be quite fast.
This way you can build entire projects, with thousands of nodes and many flows
I would also disable the mini map or stop it's events synching in near real time by throttling because it hammers performance.
So in short

Add function to set the tabs (li), set it as active, and json import that module to that tab
Switching tabs (click/change - li) clean up current tab/updates , remove drawflow (clear), allow tab the switch and use eventing, the new tab becomes active, and loads its matching module name from the editor.drawflow.drawflow[module] using the tab name text as the reference key
Import the flow from that module into that tab

Hope this helps


[WIP] Win build 7.0.2 with cmake

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

It seems as if upstream is not quite ready for windows yet. If someone would like to pick up from here, please let me know and I can provide access to my fork.

@conda-forge-admin, please rerender

Hi @traversaro, apologies for bothering you! I am trying to get scotch to build with Windows, and I got quite far. Indeed, the static build succeeds, but as you know, in conda-forge we prefer shared libs.
The problem is that by default no symbols are exported. Most issues can be fixed by CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS. But there are a few static variables that need manual fixing (see build error in https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=668453&view=logs&jobId=a70f640f-cc53-5cd3-6cdc-236a1aa90802&j=a70f640f-cc53-5cd3-6cdc-236a1aa90802&t=f5d15007-a01c-5ad8-c9ce-4d519d3b275f). I thus use GenerateExportHeader to do some magic. However, I must be doing something wrong in https://github.com/Tobias-Fischer/scotch-feedstock/blob/win-build/recipe/0016-win-fix-exports.patch. It complains that:
D:\bld\scotch-pkgs_1678401110583\work\src\libscotch\bdgraph_bipart_st.c(84): error C2099: initializer is not a constant
D:\bld\scotch-pkgs_1678401110583\work\src\libscotch\bdgraph_bipart_st.c(99): error C2099: initializer is not a constant
D:\bld\scotch-pkgs_1678401110583\work\src\libscotch\bdgraph_bipart_st.c(104): error C2099: initializer is not a constant
D:\bld\scotch-pkgs_1678401110583\work\src\libscotch\bdgraph_bipart_st.c(184): error C2099: initializer is not a constant

A Google search tells me that this is expected, as C wants to know the initializers at compile time but when using declspec this won't be known until runtime: https://www.appsloveworld.com/c/100/118/windows-struct-initialization-error-c2099-initializer-is-not-a-constant
Do you have any idea how to resolve this issue?
Many thanks,
Tobi

Do you have any idea how to resolve this issue?

Those used to develop code on POSIX are usually not aware of this issue on Windows. You cannot resolve this issue without actually changing the source code to move from compile-time initialization to runtime initialization.
Could you please paste here how the C code reads? Are these global or function-local variaables?

The relevant source code looks as follows:
https://gitlab.inria.fr/scotch/scotch/-/blob/master/src/libscotch/bdgraph_bipart_st.c#L81-84
and
https://gitlab.inria.fr/scotch/scotch/-/blob/master/src/libscotch/parser.c#L81-82
Any help would be appreciated @dalcinl

Oh, boy... there is a lot of code to fix. First, let me clarify that I'm not an expert on the windows platform. That being said, one possible hard way to fix it is to replace all initializations by constants ( eg. all pointers like &foo by NULL). And then, at some point (e.g. the many SCOTCH_*init() routines) initialize properly all the globals.
Or maybe do something like
#if defined(WINDOWS)
SomeType someglobalvar;
#else
SomeType someglobalvar = { the usual init code};
#endif

And then you add
#if defined(WINDOWS)
// some header include here, probably #include <windows.h>
BOOL WINAPI DllMain(HINSTANCE hinstDLL, DWORD fdwReason, LPVOID lpvReserved)
{
  if (fdwReason == DLL_PROCESS_ATTACH) {
    // initialize all globals here
  }
  return TRUE;
}
#endif

Years ago, at a time Open MPI supported Windows, I had to modify mpi4py sources to make it work in that platform.
Maybe there are easier ways to do it. I would do some Googling first.

Do you have any idea how to resolve this issue?

I far from keyboard until Monday, I will check this on Monday!

The first option fail with:
[242/555] Building C object src\libscotch\CMakeFiles\ptscotch.dir\bdgraph_bipart_ex.c.obj
[243/555] Building C object src\libscotch\CMakeFiles\ptscotch.dir\bdgraph_bipart_ml.c.obj
[244/555] Building C object src\libscotch\CMakeFiles\ptscotch.dir\bdgraph_bipart_sq.c.obj
[245/555] Building C object src\libscotch\CMakeFiles\ptscotch.dir\bdgraph_bipart_st.c.obj
FAILED: src/libscotch/CMakeFiles/ptscotch.dir/bdgraph_bipart_st.c.obj
D:\bld\scotch-pkgs_1678725260680\_build_env\Library\bin\clang-cl.exe  /nologo -DCOMMON_OS_WINDOWS -DCOMMON_PTHREAD -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_PATCHLEVEL_NUM=3 -DSCOTCH_PTHREAD -DSCOTCH_RELEASE_NUM=0 -DSCOTCH_RENAME -DSCOTCH_VERSION_NUM=7 -Dptscotch_EXPORTS -Drestrict=__restrict -ID:\bld\scotch-pkgs_1678725260680\work\src\libscotch -ID:\bld\scotch-pkgs_1678725260680\work\build\src\libscotch -ID:\bld\scotch-pkgs_1678725260680\work\build\src\include -imsvcD:\bld\scotch-pkgs_1678725260680\_h_env\Library\include /DWIN32 /D_WINDOWS /W3 /MD /O2 /Ob2 /DNDEBUG -DSCOTCH_PTSCOTCH /showIncludes /Fosrc\libscotch\CMakeFiles\ptscotch.dir\bdgraph_bipart_st.c.obj /Fdsrc\libscotch\CMakeFiles\ptscotch.dir\ -c -- D:\bld\scotch-pkgs_1678725260680\work\src\libscotch\bdgraph_bipart_st.c
D:\bld\scotch-pkgs_1678725260680\work\src\libscotch\bdgraph_bipart_st.c(84,37):  error: initializer element is not a compile-time constant
} bdgraphbipartstdefaultbd = { { 3, &stratdummy } };
                                    ^~~~~~~~~~~
D:\bld\scotch-pkgs_1678725260680\work\src\libscotch\bdgraph_bipart_st.c(99,59):  error: initializer element is not a compile-time constant
} bdgraphbipartstdefaultml = { { 5, 1000, 2, 10000, 0.8L, &stratdummy, &stratdummy, &stratdummy} };
                                                          ^~~~~~~~~~~
D:\bld\scotch-pkgs_1678725260680\work\src\libscotch\bdgraph_bipart_st.c(104,34):  error: initializer element is not a compile-time constant
} bdgraphbipartstdefaultsq = { { &stratdummy } };
                                 ^~~~~~~~~~~
D:\bld\scotch-pkgs_1678725260680\work\src\libscotch\bdgraph_bipart_st.c(187,33):  error: initializer element is not a compile-time constant
                                (void *) &bgraphbipartststratab },
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4 errors generated.
[246/555] Building C object src\libscotch\CMakeFiles\ptscotch.dir\bdgraph_bipart_zr.c.obj
[247/555] Linking C shared library bin\scotch.dll

I am not sure if this happens because due to the use of flang we need to use an old clang on Windows. I will try the second option.

The second option fails because in the scotch headers there is an attribute called class:
2023-03-13T17:58:37.8611350Z [249/555] Building CXX object src\libscotch\CMakeFiles\ptscotch.dir\bdgraph_bipart_st.c.obj
2023-03-13T17:58:37.8613116Z FAILED: src/libscotch/CMakeFiles/ptscotch.dir/bdgraph_bipart_st.c.obj
2023-03-13T17:58:37.8616326Z D:\bld\scotch-pkgs_1678729921646\_build_env\Library\bin\clang-cl.exe  /nologo -TP -DCOMMON_OS_WINDOWS -DCOMMON_PTHREAD -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_PATCHLEVEL_NUM=3 -DSCOTCH_PTHREAD -DSCOTCH_RELEASE_NUM=0 -DSCOTCH_RENAME -DSCOTCH_VERSION_NUM=7 -Dptscotch_EXPORTS -Drestrict=__restrict -ID:\bld\scotch-pkgs_1678729921646\work\src\libscotch -ID:\bld\scotch-pkgs_1678729921646\work\build\src\libscotch -ID:\bld\scotch-pkgs_1678729921646\work\build\src\include -imsvcD:\bld\scotch-pkgs_1678729921646\_h_env\Library\include /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MD /O2 /Ob2 /DNDEBUG -DSCOTCH_PTSCOTCH /showIncludes /Fosrc\libscotch\CMakeFiles\ptscotch.dir\bdgraph_bipart_st.c.obj /Fdsrc\libscotch\CMakeFiles\ptscotch.dir\ -c -- D:\bld\scotch-pkgs_1678729921646\work\src\libscotch\bdgraph_bipart_st.c
2023-03-13T17:58:37.8619490Z In file included from D:\bld\scotch-pkgs_1678729921646\work\src\libscotch\bdgraph_bipart_st.c:66:
2023-03-13T17:58:37.8621584Z D:\bld\scotch-pkgs_1678729921646\work\src\libscotch/arch.h(176,29):  error: expected member name or ';' after declaration specifiers
2023-03-13T17:58:37.8623687Z   const ArchClass *         class;                /*+ Pointer to architecture class         +*/
2023-03-13T17:58:37.8625514Z   ~~~~~~~~~~~~~~~           ^
2023-03-13T17:58:37.8627421Z D:\bld\scotch-pkgs_1678729921646\work\src\libscotch\bdgraph_bipart_st.c(112,56):  warning: ISO C++11 does not allow conversion from string literal to 'char *' [-Wwritable-strings]
2023-03-13T17:58:37.8629529Z                               { BDGRAPHBIPARTSTMETHBD, "b",  bdgraphBipartBd, &bdgraphbipartstdefaultbd },
2023-03-13T17:58:37.8636795Z                                                        ^
2023-03-13T17:58:37.8639502Z D:\bld\scotch-pkgs_1678729921646\work\src\libscotch\bdgraph_bipart_st.c(112,62):  error: cannot initialize a member subobject of type 'int (*)()' with an lvalue of type 'int (Bdgraph *const, const BdgraphBipartBdParam *const)' (aka 'int (Bdgraph_ *const, const BdgraphBipartBdParam_ *const)'): different number of parameters (0 vs 2)
2023-03-13T17:58:37.8642085Z                               { BDGRAPHBIPARTSTMETHBD, "b",  bdgraphBipartBd, &bdgraphbipartstdefaultbd },
2023-03-13T17:58:37.8644060Z                                                              ^~~~~~~~~~~~~~~
2023-03-13T17:58:37.8644676Z D:\bld\scotch-pkgs_1678729921646\work\src\libscotch/module.h(668,37):  note: expanded from macro 'bdgraphBipartBd'
2023-03-13T17:58:37.8645289Z #define bdgraphBipartBd             SCOTCH_NAME_INTERN (bdgraphBipartBd)
2023-03-13T17:58:37.8645989Z                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-03-13T17:58:37.8646674Z D:\bld\scotch-pkgs_1678729921646\work\src\libscotch/module.h(247,37):  note: expanded from macro 'SCOTCH_NAME_INTERN'
2023-03-13T17:58:37.8647378Z #define SCOTCH_NAME_INTERN(f)       SCOTCH_NAME_MACRO3 (SCOTCH_NAME_PREFIX_INTERN,f,SCOTCH_NAME_SUFFIXC)
2023-03-13T17:58:38.0174032Z                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-03-13T17:58:38.0321940Z D:\bld\scotch-pkgs_1678729921646\work\src\libscotch/module.h(246,37):  note: expanded from macro 'SCOTCH_NAME_MACRO3'
2023-03-13T17:58:38.0322998Z #define SCOTCH_NAME_MACRO3(p,n,s)   SCOTCH_NAME_GLUE3 (p,n,s)
2023-03-13T17:58:38.0323501Z                                     ^~~~~~~~~~~~~~~~~~~~~~~~~
2023-03-13T17:58:38.0324049Z D:\bld\scotch-pkgs_1678729921646\work\src\libscotch/module.h(244,37):  note: expanded from macro 'SCOTCH_NAME_GLUE3'
2023-03-13T17:58:38.0324688Z #define SCOTCH_NAME_GLUE3(p,n,s)    p##n##s
2023-03-13T17:58:38.0342547Z                                     ^~~~
2023-03-13T17:58:38.0343270Z <scratch space>(127,1):  note: expanded from here
2023-03-13T17:58:38.0343738Z _SCOTCHbdgraphBipartBd

I guess we are back to @dalcinl suggestions.

Thanks for highlighting the different options @traversaro @dalcinl! It seems like any of them will involve a considerable amount of work - unfortunately I won't have time to look into this; getting Windows to compile with the static builds already was a reasonable effort. I opened an issue upstream to make them aware of the conda-forge efforts: https://gitlab.inria.fr/scotch/scotch/-/issues/23
I agree with @traversaro's suggestion that a static library might be the best way forward for now until the remaining issues are fixed upstream. Do you have any objections/input @conda-forge/scotch? If not, I will revert back to the static build and mark the pull request as ready for review.

I guess we are back to @dalcinl suggestions.

Now I realize I should have been more explicit/informative to prevent all this work you did. This is not a matter of C/C++ language or compiler versions or implementations. It is all about Window DLLs and the way DLL loading works in the Windows platform. Resistance is futile.

https://gitlab.inria.fr/scotch/scotch/-/issues/23

Let see what they say. We would really need the help of the core developers to change the code, otherwise it is easy to mess up and endless to debug if all what we have are remote builds on a CI VM.

This is now ready for review @conda-forge/scotch

Kind reminder @conda-forge/scotch

@Tobias-Fischer Could you please confirm whether this PR is ready to merge?


White lines on top and bottom in full screen mode

While watching https://spee.ch/e/heli.mp4 I've noticed a white line on the top and bottom of the screen. Tom sees only one on top, Chris sees both. I'm using Chrome on Win 10.

I've also noticed that on videos with these proportions, the 'hosted via spee.ch' link gets covered by the video

@LavRadis the white line issue you are describing is when the video is full-screened, correct?

@billbitt  yes.


youtube-dl version 2019.06.27

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2019.07.02
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://soundcloud.com/aloegarten/sets/cr2015']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2019.06.27
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: ffmpeg N-94099-gdd662bbdd2, ffprobe N-94099-gdd662bbdd2
[debug] Proxy map: {}
[soundcloud:set] aloegarten/sets/cr2015: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjzs7t34q\build\youtube_dl\extractor\common.py", line 627, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjzs7t34q\build\youtube_dl\YoutubeDL.py", line 2227, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

Description

500 not fixed on newest ver

What's the link to the correct non-duplicate issue for this?

Found it, thanks.
https://github.com/ytdl-org/youtube-dl/issues/21583


Bug: Discrepancy between lint failures in autocorrect and force_update?

My CI failed with a RuboCop violation.  So locally I ran:
❯ bundle exec rake rubocop_gradual:autocorrect
Running RuboCop Gradual...
Inspecting 181 file(s) for autocorrection...
.....................................................................................................................................................................................
Fixed 0 file(s).
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................
Found 258 files with 5513 issue(s).
Processing results...
RuboCop Gradual got no changes.
noglob bundle exec rake rubocop_gradual:autocorrect  53.01s user 8.48s system 228% cpu 26.863 total

This is a surprising and invalid result, since I saw the result of running rubocop on CI (which runs check).
Instead of saying it "got no changes" it should have reported the new failures we are about to see below...
So I ran force_update:
❯ bundle exec rake rubocop_gradual:force_update
Running RuboCop Gradual...
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................
Found 258 files with 5515 issue(s).
Processing results...
Uh oh, RuboCop Gradual got worse:
-> spec/channels/modified_relays_channel_spec.rb (1 new issues)
    (line 7) "Betterment/UnscopedFind: Records are being retrieved directly using user input.
Please query for the associated record in a way that enforces authorization (e.g. "trust-root chaining").

INSTEAD OF THIS:
Post.find(params[:post_id])

DO THIS:
current_user.posts.find(params[:post_id])

See here for more information on this error:
https://github.com/Betterment/betterlint/blob/main/README.md#bettermentunscopedfind
 (https://github.com/Betterment/betterlint#bettermentunscopedfind)"
-> spec/channels/sensor_streams_channel_spec.rb (1 new issues)
    (line 7) "Betterment/UnscopedFind: Records are being retrieved directly using user input.
Please query for the associated record in a way that enforces authorization (e.g. "trust-root chaining").

INSTEAD OF THIS:
Post.find(params[:post_id])

DO THIS:
current_user.posts.find(params[:post_id])

See here for more information on this error:
https://github.com/Betterment/betterlint/blob/main/README.md#bettermentunscopedfind
 (https://github.com/Betterment/betterlint#bettermentunscopedfind)"
Force updating lock file...
noglob bundle exec rake rubocop_gradual:force_update  20.56s user 4.62s system 226% cpu 11.099 total

The force_update caught the change somehow, when the autocorrect did not.

Autocorrect should run the complete rubocop linting.  It should still fail, even if it can't fix anything.  Regular rubocop fails the same way with or without the -a flag.

@skryukov any thoughts on this?

Hey, @pboling! Sorry for the delay.

Autocorrect should run the complete rubocop linting. It should still fail, even if it can't fix anything.

Yes, that's exactly our approach—we initially run autocorrect, followed by the default rubocop-gradual command: https://github.com/skryukov/rubocop-gradual/blob/dd23f28876f045bf640a07a2a3c8844b517165ea/lib/rubocop/gradual/commands/autocorrect.rb#L10-L21
Based on the logs you shared, the number of inspected files is consistent, yet the number of detected issues varies, the question is why 🤔
I tried to reproduce the issue, but failed. To get more info we could try adding a debug option to the Rakefile. This will add awful lot of stuff to the output, including RuboCop debug information:
RuboCop::Gradual::RakeTask.new.tap do |t|
  t.options = ["--debug"]
end

One thing I have noticed is that most of the issues I've had with discrepancies have come from Betterment/betterlint or from various Bundler rules (not sure which gem they come from).  I'll add debug and see if it sheds any more light.

IMO - it is important that autocorrect run the same rules as a regular check, because that seems to be the behavior of RuboCop, and developers have been trained over many years to just run rubocop -a, and have the expectation that if there are no failures then they can push and expect it to pass the build.
Since gradual allows hijacking of rubocop to run rubocop_gradual in it's place, it makes sense to support the existing expectations and workflow of developers.

I was confused about your comment:

Yes, that's exactly our approach—we initially run autocorrect, followed by the default rubocop-gradual command:

You're saying that this is what you expect to happen for everyone, as that is what the code in rubocop_gradual is intended to do?
That is definitely nothing like our experience, if so.

More info, just saw this again, and this time (thankfully) it is in a fully open source repo.
❯ brake rubocop_gradual:check
Running RuboCop Gradual...
...................................
Found 1 files with 6 issue(s).
Processing results...
RuboCop Gradual got no changes.
❯ brake rubocop_gradual:autocorrect
Running RuboCop Gradual...
Inspecting 34 file(s) for autocorrection...
..................................
Fixed 0 file(s).
...................................
Found 1 files with 6 issue(s).
Processing results...
Uh oh, RuboCop Gradual got worse:
-> bin/bundle (1 new issues)
    (line 64) "ThreadSafety/InstanceVariableInClassMethod: Avoid instance variables in class methods."
RuboCop Gradual failed!
❯ brake rubocop_gradual:force_update
Running RuboCop Gradual...
...................................
Found 1 files with 6 issue(s).
Processing results...
RuboCop Gradual got no changes.

https://github.com/VitalConnectInc/rack-openid2

@skryukov
Just ran into this again on another private project.  I tried the debug idea you posted earlier.
Starting with a clean git working tree. I ran force_udpate to make sure I had no uncommitted changes initially, then check, then autocorrect, and what I found was very interesting.
My ENV
force_update
❯ bundle exec rake rubocop_gradual:force_update
Running RuboCop Gradual...
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Found 264 files with 3896 issue(s).
Processing results...
RuboCop Gradual got no changes.
Running RuboCop Gradual...
Gradual mode: force_update
...
{:fixed=>0, :moved=>0, :new=>0, :unchanged=>3896, :left=>3896}
RuboCop Gradual got no changes.
Finished Gradual processing in 5.077933999942616 seconds
noglob bundle exec rake rubocop_gradual:force_update  37.51s user 7.49s system 257% cpu 17.451 total

❯ git status
On branch main

nothing to commit, working tree clean

check
❯ bundle exec rake rubocop_gradual:check
Running RuboCop Gradual...
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Found 264 files with 3896 issue(s).
Processing results...
RuboCop Gradual got no changes.
Running RuboCop Gradual...
Gradual mode: check
...
{:fixed=>0, :moved=>0, :new=>0, :unchanged=>3896, :left=>3896}
RuboCop Gradual got no changes.
Finished Gradual processing in 5.030246999929659 seconds
noglob bundle exec rake rubocop_gradual:check  38.42s user 8.82s system 263% cpu 17.951 total

❯ git status
On branch main

nothing to commit, working tree clean

autocorrect
Running RuboCop Gradual...
Inspecting 338 file(s) for autocorrection...
..................................................................................................................................................................................................................................................................................................................................................
Fixed 0 file(s).
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Found 264 files with 3890 issue(s).
Processing results...
RuboCop Gradual got 6 issue(s) fixed, 3890 left. Keep going!
Running RuboCop Gradual...
Gradual mode: update
Inspecting 338 file(s) for autocorrection...
...
{:fixed=>0, :moved=>0, :new=>0, :unchanged=>3890, :left=>3890}
RuboCop Gradual got no changes.
Finished Gradual processing in 5.127219000016339 seconds
noglob bundle exec rake rubocop_gradual:autocorrect  98.46s user 15.90s system 223% cpu 51.254 total

❯ git status
On branch main
...

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   .rubocop_gradual.lock

no changes added to commit (use "git add" and/or "git commit -a")

This is where the discrepancy occurs.  The autocorrect output says there were no changes... but that's not true.
{:fixed=>0, :moved=>0, :new=>0, :unchanged=>3890, :left=>3890}
RuboCop Gradual got no changes.

The lockfile was changed.  It should have failed, but it reported success with not changes found despite making actual changes.
This is born out by running check a 2nd time, as this time it will note the failure.
2nd check
Running RuboCop Gradual...
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Found 264 files with 3896 issue(s).
Processing results...
Uh oh, RuboCop Gradual got worse:
...
Unexpected Changes!

RuboCop Gradual lock file is outdated, to fix this message:
- Run `rubocop-gradual` locally and commit the results, or
- EVEN BETTER: before doing the above, try to fix the remaining issues in those files!

`.rubocop_gradual.lock` diff:

-    [153, 9, 3, "Rails/OutputSafety: Tagging a string as html safe may be a security risk.", 193432545],
-    [171, 5, 59, "Betterment/UnscopedFind: Records are being retrieved directly using user input.\nPlease query for the associated record in a way that enforces authorization (e.g. \"trust-root chaining\").\n\nINSTEAD OF THIS:\nPost.find(params[:post_id])\n\nDO THIS:\ncurrent_user.posts.find(params[:post_id])\n\nSee here for more information on this error:\nhttps://github.com/Betterment/betterlint/blob/main/README.md#bettermentunscopedfind\n (https://github.com/Betterment/betterlint#bettermentunscopedfind)",<PHONE_NUMBER>],
-    [181, 5, 54, "Betterment/UnscopedFind: Records are being retrieved directly using user input.\nPlease query for the associated record in a way that enforces authorization (e.g. \"trust-root chaining\").\n\nINSTEAD OF THIS:\nPost.find(params[:post_id])\n\nDO THIS:\ncurrent_user.posts.find(params[:post_id])\n\nSee here for more information on this error:\nhttps://github.com/Betterment/betterlint/blob/main/README.md#bettermentunscopedfind\n (https://github.com/Betterment/betterlint#bettermentunscopedfind)", 113652520]
+    [153, 9, 3, "Rails/OutputSafety: Tagging a string as html safe may be a security risk.", 193432545]
-    [161, 9, 3, "Rails/OutputSafety: Tagging a string as html safe may be a security risk.", 193432545],
-    [186, 5, 59, "Betterment/UnscopedFind: Records are being retrieved directly using user input.\nPlease query for the associated record in a way that enforces authorization (e.g. \"trust-root chaining\").\n\nINSTEAD OF THIS:\nPost.find(params[:post_id])\n\nDO THIS:\ncurrent_user.posts.find(params[:post_id])\n\nSee here for more information on this error:\nhttps://github.com/Betterment/betterlint/blob/main/README.md#bettermentunscopedfind\n (https://github.com/Betterment/betterlint#bettermentunscopedfind)",<PHONE_NUMBER>],
-    [196, 5, 54, "Betterment/UnscopedFind: Records are being retrieved directly using user input.\nPlease query for the associated record in a way that enforces authorization (e.g. \"trust-root chaining\").\n\nINSTEAD OF THIS:\nPost.find(params[:post_id])\n\nDO THIS:\ncurrent_user.posts.find(params[:post_id])\n\nSee here for more information on this error:\nhttps://github.com/Betterment/betterlint/blob/main/README.md#bettermentunscopedfind\n (https://github.com/Betterment/betterlint#bettermentunscopedfind)", 113652520]
+    [161, 9, 3, "Rails/OutputSafety: Tagging a string as html safe may be a security risk.", 193432545]
-    [62, 9, 3, "Rails/OutputSafety: Tagging a string as html safe may be a security risk.", 193432545],
-    [77, 5, 63, "Betterment/UnscopedFind: Records are being retrieved directly using user input.\nPlease query for the associated record in a way that enforces authorization (e.g. \"trust-root chaining\").\n\nINSTEAD OF THIS:\nPost.find(params[:post_id])\n\nDO THIS:\ncurrent_user.posts.find(params[:post_id])\n\nSee here for more information on this error:\nhttps://github.com/Betterment/betterlint/blob/main/README.md#bettermentunscopedfind\n (https://github.com/Betterment/betterlint#bettermentunscopedfind)",<PHONE_NUMBER>]
+    [62, 9, 3, "Rails/OutputSafety: Tagging a string as html safe may be a security risk.", 193432545]
-    [229, 5, 3, "Rails/OutputSafety: Tagging a string as html safe may be a security risk.", 193432545],
-    [262, 5, 54, "Betterment/UnscopedFind: Records are being retrieved directly using user input.\nPlease query for the associated record in a way that enforces authorization (e.g. \"trust-root chaining\").\n\nINSTEAD OF THIS:\nPost.find(params[:post_id])\n\nDO THIS:\ncurrent_user.posts.find(params[:post_id])\n\nSee here for more information on this error:\nhttps://github.com/Betterment/betterlint/blob/main/README.md#bettermentunscopedfind\n (https://github.com/Betterment/betterlint#bettermentunscopedfind)", 113652520]
+    [229, 5, 3, "Rails/OutputSafety: Tagging a string as html safe may be a security risk.", 193432545]

RuboCop Gradual failed!
noglob bundle exec rake rubocop_gradual:check  19.91s user 4.76s system 211% cpu 11.689 total

2nd autocorrect
Strangely, and as evidence of the discrepancy, if I run autocorrect again (still with the changed lockfile that just failed the 2nd check above), it will pass saying there are no changes.
This means that check and autocorrect are not parsing rules config, or the lockfile, or both, the same way.
Running RuboCop Gradual...
Inspecting 338 file(s) for autocorrection...
..................................................................................................................................................................................................................................................................................................................................................
Fixed 0 file(s).
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Found 264 files with 3890 issue(s).
Processing results...
RuboCop Gradual got no changes.
Running RuboCop Gradual...
Gradual mode: update
Inspecting 338 file(s) for autocorrection...
...
{:fixed=>0, :moved=>0, :new=>0, :unchanged=>3890, :left=>3890}
RuboCop Gradual got no changes.
Finished Gradual processing in 5.472328000003472 seconds
noglob bundle exec rake rubocop_gradual:autocorrect  107.44s user 17.65s system 214% cpu 58.262 total

Because of this discrepancy I can no longer use autocorrect on this project, since it will always fail check in CI.

Thanks @pboling I'll try to look into it next week

I should add that so far all of the discrepancy in rules that I've paid enough attention to remember has been from one single rule:
Betterment/UnscopedFind
It comes from the betterment gem, which is an extension of RuboCop rules specifically for Rails projects.
It is possible that it is an issue of interplay between various plugins, as I do load many rubocop-* gems.

Oh!  I think I figured it out partially.  I had been referencing Betterment rules in my local rubocop config file, but had failed to use the include_gem directive.
inherit_gem:
  betterlint:
    - config/default.yml

Now that I've added that, it waffles between running force_update and check and autocorrect consistently giving the same result, and, without changing any code, giving inconsistent results.
I think a bug in the unscoped find rule may be to blame.  Perhaps it is not deterministic?
https://github.com/Betterment/betterlint/issues/51


Convert to python2/3 Compatible

@ralic is this PR still  maintained?

I created a separate repository for python 3 .
https://github.com/Tanganelli/CoAPthon3


Typo in sample code in Waypoint Update the Properties of JS Object #4438

removed extra comma

Made a mistake and put two separate issues in the one commit. Will rectify and resubmit


Update components-edge-cases.md

Add other example on how to use Webpack's asynchronous import.

Thanks for the PR! Unfortunately, I don't think the additional snippet adds much value in this context.

@phanan yeah, you are right. Thanks for reviewing my PR. I added this example because the issue I found is that it seems that when converting the name to match the html tag is not using the name property in the component, instead is converting the name set there to kebab-case.
I realized this when I named the import TFContents , then the html tag should be t-f-contents  even if the component name is tf-contents .
This is way I think is better to use the kebab-case notation when doing Webpack async import


[Front-End] Create a Home Page(index.html) to have the header + footer + body parts｛including： hobby part; event part; blogs part｝

main page UI done

ui is implemented in this pull request (not connected to backend yet).

implementated.


fix: use RDFI in batch header for returns/corrections/etc

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 59.35%. Comparing base (cb47a01) to head (b0d8d41).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #209      +/-   ##
==========================================
+ Coverage   59.30%   59.35%   +0.05%
==========================================
  Files          16       16
  Lines         779      780       +1
==========================================
+ Hits          462      463       +1
  Misses        265      265
  Partials       52       52

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


PropertyName="BodyAsHTML" usage requires additional info

Using the XML manifest from the Outlook-Add-in-Contextual-Regex example for testing, when changing the ItemHasRegularExpressionMatch Rules from BodyAsPlaintext to BodyAsHTML outlook throws an error:

These are the only change made to the XML manifest.
<Rule xsi:type="RuleCollection" Mode="And">
   <Rule xsi:type="ItemIs" ItemType="Message" FormType="Read" />
   <Rule xsi:type="ItemHasRegularExpressionMatch" RegExName="9Digits" RegExValue="\d{9}" PropertyName="BodyAsHTML"/>
 </Rule>

<Rule xsi:type="RuleCollection" Mode="And">
  <Rule xsi:type="ItemIs" ItemType="Message" />
  <Rule xsi:type="ItemHasRegularExpressionMatch" RegExName="OrderNumber" RegExValue="CO-\d{9}" PropertyName="BodyAsHTML"/>
</Rule>

What is the "highlight attribute" it is referencing?  It doesn't seem to be mentioned in the docs.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 64d9cc55-9781-d046-d612-15262d8d618e
Version Independent ID: 8c98758c-3873-b3a4-742a-8cfbd581e56c
Content: Use regular expression activation rules to show an Outlook Add-in
Content Source: docs/add-ins/use-regular-expressions-to-show-an-outlook-add-in.md
Product: outlook
GitHub Login: @jasonjoh
Microsoft Alias: jasonjoh

It's probably referring to the Highlight attribute in the rule manifest mentioned here.
When changing to
- <Rule xsi:type="ItemHasRegularExpressionMatch" RegExName="9Digits" RegExValue="\d{9}" PropertyName="BodyAsHTML" Highlight="all"/>
- <Rule xsi:type="ItemHasRegularExpressionMatch" RegExName="OrderNumber" RegExValue="CO-\d{9}" PropertyName="BodyAsHTML" Highlight="all"/>

Outlook then complains

@butchmarshall thanks for reporting this issue; we'll investigate and let you know the outcome.

@exextoc upon preliminary investigation, I suspect this may be a product bug -- can you please investigate and respond here on this thread with what you find?
The question is this: how should the manifest be structured such it will validate successfully (i.e., can be successfully sideloaded into Outlook on the web) when it contains an ItemHasRegularExpressionMatch rule with PropertyName="BodyAsHtml"?
Repro steps

Clone the Outlook-Add-in-Contextual-Regex sample repo.

Follow instructions in the repo's README to deploy the add-in's files to a web server and then update the contoso-order-number-manifest.xml file (locally, within your cloned repo) to contain the proper web server URL for the deployed add-in.

Open the contoso-order-number-manifest.xml file, replace each instance of BodyAsPlaintext with BodyAsHTML and save the file.

Attempt to sideload the add-in in Outlook on the web.

You'll get this error message when you attempt to install the custom add-in by pointing to your manifest file:

Within the Rule element on line 76 of the contoso-order-number-manifest.xml file, add the Highlight="none" attribute and save the file:
<Rule xsi:type="ItemHasRegularExpressionMatch" RegExName="OrderNumber" RegExValue="CO-\d{9}" PropertyName="BodyAsHTML" Highlight="none" />

Attempt to sideload the add-in in Outlook on the web.

Once again, you'll get this error message when you attempt to install the custom add-in by pointing to your manifest file:

Note: I wouldn't expect to receive this error message, given that the docs state the  Highlight attribute is only relevant to Rule elements within ExtensionPoint elements and the manifest is indeed specifying the Highlight attribute for one of the Rule elements within the And RuleCollection (in a Rule element within a ExtensionPoint element) .

Within the Rule element on line 76 of the contoso-order-number-manifest.xml file, change the value of the Highlight attribute to "all" and save the file:
<Rule xsi:type="ItemHasRegularExpressionMatch" RegExName="OrderNumber" RegExValue="CO-\d{9}" PropertyName="BodyAsHTML" Highlight="all" />

Attempt to sideload the add-in in Outlook on the web.

You'll get this error message when you attempt to install the custom add-in by pointing to your manifest file:

It's not clear how to fix this dilemma: Outlook seems to require the Highlight attribute in this location, but does not seem to acknowledge Highlight="none" and does not allow setting Highlight="all" when PropertyName="BodyAsHTML".

BodyAsHTML does not support Highlight = all, due to the fact that parts of the HTML body may not be able to be highlighted by the e-mail. For example, the  tag or  etc. So that gets rejected. If you leave off highlight = all, it will default to highlight = none, and then the rules won't have a highlight so the manifest will be rejected.
BodyAsPlaintext should be used as the highlight rule to get around this.

@exextoc thanks for the clarification about this behavior.
@butchmarshall -- as @exextoc describes, you must set PropertyName="BodyAsPlaintext" if you want to use the Highlight attribute.  We'll update the docs soon to reflect this info:

https://docs.microsoft.com/en-us/outlook/add-ins/use-regular-expressions-to-show-an-outlook-add-in#using-regular-expression-results-in-code
https://docs.microsoft.com/en-us/office/dev/add-ins/reference/manifest/rule#itemhasregularexpressionmatch-rule

Tracking doc update internally with: https://office.visualstudio.com/OC/_workitems/edit/2716227.

The two articles I referenced in my previous comment will be updated with this information when doc updates are published later this week.

So this means we cannot use BodyAsHTML PropertyName with ItemHasRegularExpressionMatch at all, right? What if I need to find an anchor tag (<A href=""..) or DIV?


Unclear how to define "initial" and "reset" img coordinates?

From the documentation, it is not clear to me how i can set the initial coordinates for an image to initialize on.. and furthermore, how to set those coords to be the "reset" coords?

See the startTransform option.
$elem.panzoom({
  startTransform: 'translate(30)'
});


[SDK] zunion command missing in SDK

The @upstash/redis SDK seems to be missing the zunion command.
SDK Version: 1.22.0

I would like to use zunion/zintersect too.

/bounty $50

/bounty $50

/attempt #473


[Table] Default Pagination Styles With "small" Table

What problem does this feature solve?
At the moment, if we set nzTableSize as small, then the "mini" pagination is used. It would be great for user experience if we can have an option to set table pagination styles as well as an @Input
What does the proposed API look like?
nzPaginationType="paginationType"
where "paginationType" is 'default' | 'mini' | 'normal'

If nzPaginationType is default and nzTableSize is small, will the final result be small?

If nzPaginationType is default and nzTableSize is small, will the final result be small?

default pagination type will result in the same behavior as of today. We will need a new pagination type e.g. normal which will keep the pagination buttons of the same size as with default table size.

Hello faisalmuhammad, I think your suggestion is very good, but I think it is more appropriate to only set default and small for nzPaginationType. If it is default, then it will be based on the nzSize of the table as before. If small is set, then nzPagination will become a small size.


Extended fork

Hello, I made a fork of this library to write a supertinycron tool:
git@github.com:exander77/supertinycron.git
I extended the library for expressions described here:
https://www.javadoc.io/doc/org.quartz-scheduler/quartz/latest/org/quartz/CronExpression.html
https://github.com/gorhill/cronexpr
Would you be willing to take a very large pull request?

Hi, unfortunately I don't use ccronexpr myself anymore and cannot maintain it properly. Would it be convenient if I archive this repo and add a link to your repo on top of the README (instead of accepting the PR)?

@staticlibs I wrote supertinycron based on your work, and it is working great so far.
I think I fixed all the issues and implemented Quartz syntax as was requested here.
I also fixed building using Gihtub actions in my repo, and it should be working as was in your original repo. You can review here: https://github.com/exander77/supertinycron/actions
So my repository should behave as your original one compiling under -std=c89, -std=c++11 in gcc and clang, static and non-static. I don't use it on embedded hardware, so I can't really test that, but I don't think I interfered with any of the code that handles that. I added year functionality that consumes an additional 29 bytes for year bit field and I may introduce some compile macro to disable for the embedded hardware purposes.
If it is ok with you, you can do as you suggest.
Btw, great work on this, I tried to follow the ideas of the original next/prev event calculation, but I merged most of the prev end next code, so it is more reliable as I primarily test next. I also implemented tonne of additional tests for new features as well as incorporated tests from this issue tracker.
I also find a bug recently, that I intend to fix as well:
60 * * * * * *
* 60 * * * * *

I wanted to find leap seconds for testing purposes, but neither glibc, nor musl handles this and ccronexpr segfaults due to not finding any matching in bitfield.

Archiving the repo now, thanks for keeping this project alive!


Add basic DOAP file for Pulsar

This patch adds a default DOAP file for Pulsar
See here for reference
https://projects.apache.org/create.html

@Anonymitaet @Jennifer88huang  @dave2wave
once we commit this file I expect it to be available at https://pulsar.apache.org/doap.rdf
and then I have to a reference to that link here:
https://svn.apache.org/repos/asf/comdev/projects.apache.org/trunk/data/projects.xml