Create icon view for users and roles

Create an icon view for:

[x] processes
[ ] users
[ ] roles

I've marked the places where the icon-view components should go with a TODO :)


sending id in POST doesn't work

Hi,
When doing POST with the id it does not work, it is not being passed and saved to mongo, mongo generates its own.
I have used this code to generate valid ObjectId
const ObjectId = (m = Math, d = Date, h = 16, s = s => m.floor(s).toString(h)) =>
    s(d.now() / 1000) + ' '.repeat(h).replace(/./g, () => s(m.random() * h))

from https://stackoverflow.com/questions/10593337/is-there-any-way-to-create-mongodb-like-id-strings-without-mongodb
In order to override it i had to update the Model Scehma (created from mongoose) of
{'_id: String} and import it in the index.js file of the Entity/Resource.
like this
router.post('/',
  ...
  body({ _id , ...})
  ...

but now it is mandatory to specify it.
I am doing this because my client generates the id.
Any ideas?
thanks.

have you tried  something like this ?
Profiles: [{
_id: false,
myCustomID: {
type: string,
index: true
},
phone: {
type: Number,
required: true
}
}],
Regards !


Improve html sanitization

Grouping issue about html sanitization.
Please link issues/PRs to this. We can have the discussion and options ( like http://htmlpurifier.org/ ) in here and derive todos from it.

Things to do

[ ] Research list of alternatives
[ ] Pro and con list of all alternatives
[ ] Select replacement
[ ] Impact analysis. Which files need changing, which operators need changing
[ ] Documentation assessment. Which parts of the documentation need to be adapted
[ ] New branch for all the neccessary changes, all changes go into that branch
[ ] Upon completion of all tasks, merge branch

Check #2520 for references.

Check https://github.com/RSS-Bridge/rss-bridge/blame/master/lib/FormatAbstract.php#L164 . logman already provided htmlpurifier as an alternative when he described the function.

I found a relevant function in lib/html.php:
/**
 * Removes unwanted tags from a given HTML text.
 *
 * @param string $html The HTML text to sanitize.
 * @param array $tags_to_remove A list of tags to remove from the DOM.
 * @param array $attributes_to_keep A list of attributes to keep on tags (other
 * attributes are removed).
 * @param array $text_to_keep A list of tags where the innertext replaces the tag
 * (i.e. `<p>Hello World!</p>` becomes `Hello World!`).
 * @return object A simplehtmldom object of the remaining contents.
 *
 * @todo Check if this implementation is still necessary, because simplehtmldom
 * already removes some of the tags (search for `remove_noise` in simple_html_dom.php).
 */
function sanitize($html,
	$tags_to_remove = array('script', 'iframe', 'input', 'form'),
	$attributes_to_keep = array('title', 'href', 'src'),
	$text_to_keep = array()){

	$htmlContent = str_get_html($html);

	foreach($htmlContent->find('*') as $element) {
		if(in_array($element->tag, $text_to_keep)) {
			$element->outertext = $element->plaintext;
		} elseif(in_array($element->tag, $tags_to_remove)) {
			$element->outertext = '';
		} else {
			foreach($element->getAllAttributes() as $attributeName => $attribute) {
				if(!in_array($attributeName, $attributes_to_keep))
					$element->removeAttribute($attributeName);
			}
		}
	}

	return $htmlContent;
}

Just so we are clear about what we are talking about here. We are talking about sanitizing user input that we know is html.
This means we want to take a string of html and explicitly keep certain html tags and html tag attributes. The rest we throw away.
The main purpose is to nicely display the feed item in a reader but to also remove harmful stuff such as:

lawls

Notice that sanitizing is different from html escaping. Html escaping means converting html to something that the browser does not interpret as html. So e.g.
kek
 => <h1>kek</h1>

This is mainly a problem in our HtmlFormat. It is very easy for feed owners to insert any kind of html/js. Luckily, feed readers do sanitizing too. So it's not really a big problem.
A related issue is that we should cleanup the data we insert in our feeds so that they render properly in feed readers.

https://github.com/tgalopin/html-sanitizer/

While this is being considered; I'd like to chime in with a use case that is greatly harmed by sanitization; so a toggle of some sort might be considered.
When using rss-bridge to generate feeds, it can often be very useful to include style tags or occasionally iframes. As the author of the programs generating the feeds' contents, I'd greatly prefer to not have content stripped out of the feeds I design.
I understand the security concerns, but needing to use rss-bridge to output garbage (including html as text etc.) and write custom plugins for feed readers to convert that garbage back to the correct content is both extremely inconvenient and extremely impactful to performance. An enormous benefit of rss-bridge is the ability to process feed contents at refresh-time rather than view-time, saving human users actual time via reduced page loads, so having to run another plugin in the reader on every refresh of every entry greatly increases load times.
Example usecases:
Using style tags on imgs to make their size shrink to fit the client's screen; which makes quickly viewing tons of images many, many, many times faster compared to needing to scroll down frequently to view offscreen content for tall/high-resolution images (when reading comics, for example!)
Using iframes to embed urls from specific trusted websites. For example, when writing a Bridge to parse Reddit posts, including gfycat links as iframes means I can use their official url format and not need to query their API and parse json to make a direct link to a video; which also means giving up UI like author name/link, tags, etc.
RSS-Bridge is a fantastically useful program, and I think that giving bridge developers the option to disable sanitization on the output they design will make it even better.


Can Gridsome do anything that Nuxt can't do?

a developer tells me gridsome can't do anything that NUXT can't do regarding static site generation. Should I believe him?

I don't think we can compare Nuxt and Gridsome based on what they can do. It's how you create a website that is the main differences between them :)

this question is little ambigous

Okay let's elaborate on that because I'm also interested as I'm only used to Nuxt and have problems to find the answers to my questions.

Can I animate between pages?
When statically generated, are page switches is the framework able to load data synchronously on client-side or will always the static file be loaded?
Can I use Gridsome with TypeScript?

We are working on a solution for page transitions.
Vue.js runs in the browser so you can load data from APIs.
I haven't tried Gridsome with Typescript, but it should work with custom webpack config.

While Nuxt is a great framework and configurable to create anything you want, Gridsome is more focused on websites that contain mostly static content. We try to solve common problems for most use cases; Source plugins make it easy to use content from CMS'es (more plugins are coming). And a GraphQL data layer lets you easily access the data in any component. All routes are rendered automatically, including dynamic routes. Pagination for collections and taxonomy pages can be implemented with just a few variables in your query. Local images are optimized automatically and lazy-loaded in the browser. All this can be achieved without writing any logical code.
You can use the Data Store API to insert data from custom APIs, but we are going to open up methods for more advanced usage before a 1.0 release.
When statically generated, the app only contains vue, vue-router and vue-meta with minimal custom code from gridsome to lazy-load images and preload data plus a few components. Vue.js hydrates the markup and turns into a SPA which allows you to do almost anything in the browser. All data is also rendered as static JSON and fetched before each route, so there is no connection with the CMS'es when generated. And only data you query for is rendered.

Like what @hjvedvik said, Gridsome tries to simplify building modern, static websites for any data source. To compare Gridsome and Nuxt you also need to compare it with nuxt generate.
The biggest difference between is probably how you work with data.
In Nuxt you fetch directly from APIs per route. This can be from an external rest API or GraphQL API.  It’s up to the developer on how to do this.
For example, to create a blog with pagination in Nuxt you’ll need to:

Create a /pages/blog/index.vue file and fetch blog posts into this from any API.
Manually create pagination functionality.
Create a /pages/blog/_id and fetch from any API by using :id route param as a variable.

The biggest problem with Nuxt (Generate), in my opinion, is that it always fetches from the API on client-side after page load, even if it's static. I know they are working on fixing this, but this is how it is now. There could, of course, be some quick fixes for this.
In Gridsome you fetch all data at build time or when starting development server into an internal GraphQL data layer and THEN you start adding data to pages from this data layer with GraphQL queries. You don't need to deal with any API endpoints.
Data source plugins are designed to make it as easy as possible to add and work with data in Gridsome. For example, if you connect to a WordPress site you can get posts, categories & tags connections out-of-the-box.
To create a blog in Gridsome you’ll need to:

Create a /pages/Blog.vue file and add a GraphQL query for the collection you want. This is super easy thanks to the GraphQL Playground where you can build and test queries.
Use the built-in pagination for the GraphQL query, so archive pages like blog/1 blog/2 etc. works.
Create a /templates/TheCollectionName.vue that will automatically be the template for the GraphQL collection.

Gridsome should also generate pages faster because of how it fetches data up front instead of per page.

thanks everyone. this is very helpful. could be some useful content to distill into docs one day!

Thanks for the details. Very helpful. I also see potential putting this comparison to the README. I still feel like these two can be compared, even generate in Nuxt is only a very very small part of its feature set. It comes with so many built in performance centered features that at least for me it needs quite something to get me hooked up to something else.

The biggest problem with Nuxt (Generate), in my opinion, is that it always fetches from the API on client-side after page load, even if it's static.

Maybe the Nuxtent Module I use in my case is intercepting this behavior, but at least it doesn't load again client-side on first load for me. Only on routing.
I guess we can say the biggest advantage of Gridsome really is the data connectors, as in Nuxt you have to do it manually or create something on your own, that works for static generation but doesn't put the whole data into your JavaScript chunks. And here it gets tricky, because that means you have to know quite something about Webpack to extend that part. Or you try to find an existing module for it but in my experience it's hard to find one that is enough extended and has frequent maintenance.
Somehow I would love to be able only using the data layer features of Gridsome, but use Nuxt for the rest. The advantage of Nuxt is that I can decide between SPA, server-side or static generation. That allows me to make a full web application that uses the best of all 3 methods in parallel e.g. shows static-generated pages for public pages but makes the switch to server-side rendering when e.g. user logs in and needs data that only belongs to the user. SPA is almost never an option for me.
Does Gridsome also work with server-side rendering of Vue.js?
Also something that turns me off a bit is that the documentation contains many empty pages. In my humble opinion, I would only put stuff in there that is already implemented and only release features when it's documented. Because that's one of the biggest strengths Vue.js and Nuxt.js come with: A very strong documentation. But I know that my concerns are probably not valid as long as it doesn't reach v1.0.0.

Somehow I would love to be able only using the data layer features of Gridsome, but use Nuxt for the rest. The advantage of Nuxt is that I can decide between SPA, server-side or static generation.

We're a strong believer in JAMstack. That means generating as much as possible up front (at build time) and then add dynamic stuff by fetching data on the client side. So I think Gridsome can be used for most things if you go this route. For example, user data could just be fetched on the client side if the user was authenticated. This makes it possible to host your app on a CDN and you don't need a Node.js server.

Does Gridsome also work with server-side rendering of Vue.js?
Gridsome is server-side rendered at built time.

Also something that turns me off a bit is that the documentation contains many empty pages.
Thanks for the feedback! This is there so people could contribute if they want. If not we'll try to add them before 1.0 is released.

Does Gridsome also work with server-side rendering of Vue.js?

As @tomtev wrote, all pages are server-side rendered by Vue.js at build time. But Gridsome also has a built-in gridsome serve command (will work again in v0.5) which will run the website on an Express server with server-side rendering. All data is fetched from the GraphQL endpoint before each route, so it's very fast to not be JAMstack. But I wouldn't recommend using it yet.

Gridsome is server-side rendered at built time.

I don't think that is what server-side rendering means. In my understanding that's just static-generation. Server-side rendering is what @hjvedvik means with the dev server.

This makes it possible to host your app on a CDN and you don't need a Node.js server.

Yes, that's nice and that's what I do for many cases, but I'll never ever just use SPA for things like user areas. A server-side or static-generated fallback is always required in my cases to ensure client-side error fallbacks, usability and accessibility workarounds.

I think I now understood that this is completely targeted in direction of the JAMStack. Still, I think your work with the data adapters is awesome and if I would have the possibility to use them outside of Gridsome for all those cases where I can't leverage the JAMStack thus need server-side rendering too, that would be a dream. But I understand that this is miles out of scope.

If we can’t agree that server-rendered at build time and static generation are the same thing, we need to just pack it all it and become sheep herders.

If we can’t agree that server-rendered at build time and static generation are the same thing, we need to just pack it all it and become sheep herders.

So then let me correct and say I just never heard it in that context and find it misleading as I clearly talk about server-side rendering as a fallback for SPA in the example context of user logins.

SSR and static rendering are related,  but not the same.
Static rendering generates a page once and then serves that same page to everyone who requests it. The content of the page is considered static and cacheable.
SSR is typically rendered on every request to the server, and typically has a low cache ttl (or never cacheable).

We talk about "Server-side rendering at build time" because that is exacly what's happening. The pages are rendered at the server, not the client. They are pre-rendered at build time instead of when a user is requesting it. This results in better speed and no need for caching etc.

They are pre-rendered at build time instead of when a user is requesting it.

That is it... A better term would be pre-rendered at build time.

This results in better speed and no need for caching etc.

No need for caching?  Caching helps definitely with static html sites, especially large ones (i.e. either local cached copies on the users browser, or cached on an intermediate cache, such as CloudFlare  etc).

Jah wouldn't now start mixing in any caching topics. It's also good practice to cache HTML and actually when you use a CDN e.g. Netlify, that will pretty sure cache things on a cache proxy server (as it's actually the point of the CDN to be fast).
Pre-rendered is exactly the right term here. Anyway, I didn't want to open Pandora's box about a SSR discussion. I got my answers and will keep Gridsome on the radar for production when it hits 1.0.0.
Keep on going with the great work. Looking forward to that animation/page transition support.

“Why did you stop developing Gridsome?”
“People kept arguing the semantics of ‘server-rendered’.”

a pretty good recent article on the topic
https://developers.google.com/web/updates/2019/02/rendering-on-the-web

If you have a large blog, will Gridsome prefetch hundreds of pages of content? Isn't that pretty heavy, especially for mobile users?
We are building a new blog with Nuxt pulled data from Airtable. Right now it's just a prototype, and I'm interested in switching to Gridsome because I've done several sites in the past with Gatsby and have a high opinion of it. But 1) Gridsome development seems to be going pretty slow, so it looks like we might not see v1 in the near future, and 2) we may have hundreds or thousands of posts over time, so I'm not sure if the prefetching will be a problem will actually hurt Google's site performance ratings. With VuePress we found we actually scored higher on performance if we disabled prefetching.
Finally, if there a way to build an "infinite scroll" experience Gridsome? For our main page, we'd like to have a chronological list of all our articles, newest first. With a SPA model I'd just load the first 10 and them implement a scroll trigger to pull more content from the API. With a static site, should I just pretty much just load all the content in one huge page, or is there a more sophisticated strategy?
@hjvedvik @tomtev
Finally, how can I support your development? is there something like Paetron in Norway?
Cheers!

@brandonpittman
Sorry about that :) Was just trying to explain what happens under the hood. Because it helps us understand how to deal with SSR plugins beacuse it is the same for normal SSR and Static Rendering.
I know Gatsby is using "Server-side rendering at build time" in their presentations.
@Maxhodges

If you have a large blog, will Gridsome prefetch hundreds of pages of content? Isn't that pretty heavy, especially for mobile users?

It only prefetches links that are in view. So it's perfect for mobile users.

Gridsome development seems to be going pretty slow, so it looks like we might not see v1 in the near future

We will release 0.5 today. It is a huge version with many important features and changes under the hood. We're actually not that far from 1.0 :)

we may have hundreds or thousands of posts over time, so I'm not sure if the prefetching will be a problem will actually hurt Google's site performance ratings.

It only prefetches links that are in view.

Finally, if there a way to build an "infinite scroll" experience Gridsome? For our main page, we'd like to have a chronological list of all our articles, newest first.

We will have a built-in feature for this in the future. Just like our Pagination support. For now you need to load all posts with GraphQL and just manually split the post array.

Finally, how can we support your efforts on Gridsome? is there something like Paetron in Norway?

We might start a Patreon or Open Collective very soon.

We'll close this now :)
Here is a great podcast about Next VS Gatsby:
https://syntax.fm/show/120/gatsby-vs-next
Next is the react alternative for Nuxt
Gatsby is the react alternative for Gridsome
Nuxt do some stuff better and simpler than Next tho :)

Hello. @hjvedvik
I noticed you are expert in Gridsome area.
Now I am developing Gridsome project.
It worked well for a while.
But for now I am getting the following error.
Please help me.

Looking forwards for your good reply.
Thanks.

Hi @petar7777. That issues has already been reported here, with suggestions for fixes:
https://github.com/gridsome/gridsome/issues/252

Hi @robaxelsen
Thanks for your quickly reply.
OK. I will follow it.
Thanks.
:+1:

@tomtev

Gridsome should also generate pages faster because of how it fetches data up front instead of per page.

Indeed that's how it's typically done, Nuxt does allow to fetch all your data before compile time, and pass the data to the pages.
ie, you could fetch all users from a database in the config's generate() function, and pass that info to pages via a payload, and use it in the async() method and pre-render that data
Both frameworks are great though.

@Maxhodges we also use Airtable as our backend, though for much more than just Blog posts. We organize weddings all over Europe, so we have 100s of venues and "projects" (past weddings) we showcase on our website, plus all you typical pages like /contact, /faq etc., as well as blog posts.
I plan to use something like Nuxt or Gridsome to pre-render 99% of the content, of which the bulk will come from Airtable records (+Cloudinary for images ...we have 1000s of them) and a few really static ones (homepage, /contact etc.) from YAML files, which my content guys will edit directly on GitHub first and later with something like Forestry (if needed). Later we might add few SPAs for things like "MyWedding" page where our clients will manage / track progress of their wedding being planned etc.
Gridsome makes really good impression and I like the idea of "source plugins" (though the Airtable plugin is no good for us, because it only gets data from single Airtable table), though after doing huge corporate ERP projects for 10 years, I always also keep other things in mind such as maturity, stability, docu, availability of freelance talent, paid support from the creators with SLA, potential longevity etc., so as of now I'm also debating Gridsome vs. Nuxt.
Having said that, it will be another 2-6 months until we actually will start implementing this project, so I'll keep an eye on Gridsome until then!
Are you on Discord?
Let's stay in touch to exchange ideas etc.
@tomtev

The biggest problem with Nuxt (Generate), in my opinion, is that it always fetches from the API on client-side after page load, even if it's static.

...and...

In Gridsome you fetch all data at build time

I'm not 100% sure, isn't this how you do it in Nuxt?
https://nuxtjs.org/api/configuration-generate/#routes
(see Function which returns a Promise)
So in other words:
Gridsome:
// gridsome.server.js
const axios = require('axios')

module.exports = function (api) {
  api.loadSource(async actions => {
    const collection = actions.addCollection('Post')

    const { data } = await axios.get('https://api.example.com/posts')

    for (const item of data) {
      collection.addNode({
        id: item.id,
        title: item.title,
        content: item.content
      })
    }
  })
}

...equals Nuxt:
nuxt.config.js

import axios from 'axios'

export default {
  generate: {
    routes () {
      return axios.get('https://my-api/users')
        .then((res) => {
          return res.data.map((user) => {
            return '/users/' + user.id
          })
        })
    }
  }
}

?
Note I am noob in the whole headless, JAMstack, SPA, ...and even JS world, so I might be totally confusing things here, only judging from reading the docus.

@peachp Nuxt isn't fully static and in my understanding, follows the way how Gatsby (correct me if I'm wrong) does it: Generate static pages for first load, but allow client-side async loading for navigation (as soon as you navigate after first load).
Nuxt allows you to do stuff like page transitions or transitions for loading and including async data. But it also requires more traffic from your API as, in addition to the build that makes API requests on every change, every user also makes API requests on the running application. This can especially be a drawback if you use something like Dato CMS or Contentful, when you have API request limits.
Gridsome is fully static and thus doesn't come with the overhead of client-side async requests. This can especially be useful if you want to hide the API you are using and want to save requests. Most of the static-site generators work the way Gridsome works.
A fully static generated mode for Nuxt is also in the works.

@peachp Nuxt isn't fully static and in my understanding, follows the way how Gatsby (correct me if I'm wrong) does it: Generate static pages for first load, but allow client-side async loading for navigation (as soon as you navigate after first load).
Nuxt allows you to do stuff like page transitions or transitions for loading and including async data. But it also requires more traffic from your API as, in addition to the build that makes API requests on every change, every user also makes API requests on the running application. This can especially be a drawback if you use something like Dato CMS or Contentful, when you have API request limits.
Gridsome is fully static and thus doesn't come with the overhead of client-side async requests. This can especially be useful if you want to hide the API you are using and want to save requests. Most of the static-site generators work the way Gridsome works.
A fully static generated mode for Nuxt is also in the works.

@renestalder
Hm... interesting. I'm quite new to the whole headless / static and even JS world, so it is not always clear that there are all these flavors of static / pre-rendered / SSR / headless and so on.
The way you describe it, it sounds like this Nuxt's generate: routes ... config is really just to generate the routes for navigation, but the data is fetched at run-time for each request (unless cached) from your API of choice?
However, when I read further, it says something about payload and generate.interval, which indicates that some data (payload) is indeed pre-fetched / pre-rendered at generate / build time, and not jsut at run-time?
Speeding up dynamic route generation with payload
In the example above, we're using the user.id from the server to generate the routes but tossing out the rest of the data. Typically, we need to fetch it again from inside the /users/_id.vue. While we can do that, we'll probably need to set the generate.interval to something like 100 in order not to flood the server with calls. Because this will increase the run time of the generate script, it would be preferable to pass along the entire user object to the context in _id.vue. We do that by modifying the code above to this: ...
Not sure if I interpret it correctly.
In any case, for our business, we would indeed need to fetch / render as much of the data and HTML etc. as possible, because our website is 95% static (content changes every few weeks) and we'll only have 2-3 SPAs.

I'm the OP, but after our team built sites with Nuxt, Gridsome, VuePress, we prefer both Gridsome and VuePress over Nuxt. I'll see if our CTO @frandiox would like to chime in

Thank everyone, quite helpful content, easy to get a big picture for someone who wants to choose either of one for their application. :)

@tomtev

We might start a Patreon or Open Collective very soon.

It seems Gridsome dev has really slowed. Is funding an issue?  Did you setup a Patreon?

@Maxhodges Money is not a problem. It's the time. We're still working a lot on Gridsome and planning how 1.0 will be. There is also a lot of work and planning happening outside of the repository. Like for example how to make sure v1.0 will be 100% ready for Vue 3.0.
i18n works different for every data source, so it's hard to make an official solution. It's best to solve i18n by just creating a new data source and setup different routes per language. I think this is how Gatsby is solving it. There is no official solution for Gatsby I think?

thanks, I tried reaching out to you guys off-repo...
is there anything I can do to help accelerate development? Maybe start an official Paetron for Gridsome and use it to incentivize other contributors?

It is very sad to see that the only Vue alternative for GatsbyJS is having such a slow progress. Not to be rude or so, in fact I am very happy to see what product it is evolving into however I believe it also misses out a lot on potential users due its lack of development. VueJS 3.0 is a big thing and I waiting for it as well. Especially for native TypeScript support but it shouldn't be a major slowdown as it will not introduce any major breaking changes. :)
By writing, I simply express my concern and interest in Gridsome. And of course, I would happy to become Patreon if that would help!


Pull metrics from DHS data for empowerment work

We need several new data points for the empowerment enhancements. I will pull them from the most recent Kenya DHS. These will include:

[x] Percentage of women with each number of years of education by age and parity
[x] Probability of doing paid work by age
[x] Average decision making power over wages by age
[x] Average ability to refuse sex by age
[x] urban/rural percentages
[x] age distribution of partnership formation

From our PMA analysis, we will need the parameters from the functions for change in the three empowerment metrics annually. These will include coefficients from the functions:

[ ] contraceptive use by age, parity, postpartum, work, control over wages, ability to refuse sex, partnership, education, and urban/rural
[ ] work by age, parity, partnership, education, and urban/rural
[ ] control over wages by age, parity, partnership, education, and urban/rural
[ ] ability to refuse sex by age, parity, partnership, education, and urban/rural

@mzimmermann-IDM: While refactoring the loading of education.csv (from  pandas multiindex dataframe to ndarray) I noticed the following:
 data["edu"].unique()
array([24, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10,  9,  8,  7,  6,
        5,  4,  3,  2,  1,  0])

There's no data for "22" and "23" education years. Is this expected?

Definitely not! I realized I calculated these wrong, and am still trying to figure out how to do it. I will let you know when I have a better version.


Map: Megastudy row-specific vocabulary endpoint

The megastudy has variables, like species, which do not have a reliable vocabulary across the entire term. Instead the vocabulary might be study or collection specific, meaning each row of some downstream entity has a different expected vocabulary based on an upstream entity. We need a mechanism to find the expected vocabulary AFTER filters have been taken into account.
This could be a new metadata-style endpoint, which other plugins could call. It would need to know which ancestor variable contains the complete row-specific vocabulary, and what filters have been applied to the child variable (ex species).
depends on https://github.com/VEuPathDB/EdaLoadingIssues/issues/63

we may hard code the study entity as being the one we always look to to infer the vocabulary

for more info, were planning to add a new annotation hasStudyDependentVocabulary. this will need incorporated into the reference metadata in edacommon presumably (dont think theres a separate ticket for that, not sure if we want one). the value of this annotation would be true/ false, and for vars where its true we need to be able to hit this endpoint which id imagine should:

accept a set of filters, and the var spec for the var in question
returns results in the entity space of the var in question, each row w a list of expected values/ vocabulary
ideally have the results meet the merge service api, in case we want to modify the data flow at some point to have the merge service automagically provide this for us. (disclaimer: idk how hard that is)
for each row in the output entity, use its upstream study entity to find all possible values for the var of interest in that study. then remove values explicitly filtered against.

I worry a little bit about merging the vocabulary rows into the entity space peformance-wise for the map. It's going to be potentially a lot of duplicate data depending on how big these vocabs can get if we're merging into a high-cardinality entity. This might be short-sighted, but part of me just wants to return the filtered mapping of study IDs to vocabularies and load that whole thing into memory and merge on the fly. This is probably a bad idea for a lot of reasons though.

No, not a bad idea. I can work w that I think without too much more work than what I'd have otherwise done. And I suspect the merge service might prefer it too.

I'd like @ryanrdoherty thoughts I think about the merge service...

Yeah, I'll raise as a topic for after our scrum tomorrow. We can pull you in as well if you're free. I need to think about this a little more; it's possible automatically doing the work in subsetting/merging service wouldn't be too bad.

I can't comment on how useful that will be to consumer services, but aren't all the counts going to be zero?  Oh, no, maybe not, they are the counts of zeroes? Who would consume those counts?
Also, I'm curious how the study-specific vocabulary is calculated. Presumably it's the same thing as the vocabulary in the study metadata? Is it pre-calculated/stored in db at load-time?

I assume what the data service plugins need is, instead of a stream of data looking something like this
2001-02-03  SpeciesB  55
2001-02-03  SpeciesC  123
2001-02-03  SpeciesE  99
2001-02-04  SpeciesB  1
2001-02-04  SpeciesD  22

it would get something like this
2001-02-03  SpeciesA  0
2001-02-03  SpeciesB  55
2001-02-03  SpeciesC  123
2001-02-03  SpeciesD  0
2001-02-03  SpeciesE  99
2001-02-04  SpeciesA  0
2001-02-04  SpeciesB  1
2001-02-04  SpeciesC  0
2001-02-04  SpeciesD  22
2001-02-04  SpeciesE  0

Thanks Bob, that makes sense. After some discussion with Danielle and Ryan, we're leaning toward an endpoint that can provide the mapping from study to vocabulary which can be used by data service to impute zeroes where necessary. This has the benefit that it should be quick to compute and won't require us to append the vocab to every row in the sample-space.

OK thanks for the update @dmgaldi - I leave it in your six capable hands.
Just checking that when we say "study" we mean the study entity in a megastudy, otherwise the whole study for non-megastudies (though there are no current plans to require special zero handling for these)

I think probably we just mean root most entity..

Yes, definitely for the megastudy. That would save hardcoding any entity ids


NOPERM this user has no permissions to run the 'info' command or its subcommand

NOPERM this user has no permissions to run the 'info' command or its subcommand (Red Alert in GUI)
I am getting this alert because the user ACL is Read only permission .I found the solution to this issue over here,
it says to disable the info call with

enableReadyCheck: false

on RedisOptions. But their is no RedisOptions to pass through this GUI so please provide such option.
Error Screenshot

Interesting. I'll see if the parts that call INFO could be disabled

Should be addressed in v1.2.12. Let me know if it is still an issue.


The client ( version 0.3.0) ignores the URL configuration parameter.

Client machine: <IP_ADDRESS>
Machine with control plane: <IP_ADDRESS>
===== Script =====
import llama_deploy
apiserver_url = 'http://<IP_ADDRESS>:4501'
async def check_status(apiserver_url):
# Pass other config settings to the Client constructor
client = llama_deploy.Client(api_server_url=apiserver_url,
timeout=10
)
status = await client.apiserver.status()
logger.info(status)
yield status
(...)
Output:
(...)
File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 761, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://localhost:8000/sessions/create'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

Hi @jdelacasa I see in the logs the error is for sessions/create but in the code you do client.apiserver.status(), could it be the error is actually in the parts of the script omitted?

yes sorry , the correct:
async def stream_events(apiserver_url, deployment_name, payload):

    client = llama_deploy.Client(api_server_url=apiserver_url,
                                timeout=10
                                )

    # Create a new session
    session = await client.core.sessions.create()

    task_id = await session.run_nowait(
        deployment_name, payload
    )

    async for event in session.get_task_result_stream(task_id):
        if "progress" in event:
            yield f'Workflow Progress: {event["progress"]}\n'

    final_result = await session.get_task_result(task_id)
    yield f'Final Result: {final_result}\n'

    # Clean up the session
    await client.core.sessions.delete(session.id)

return StreamingResponse(stream_events(apiserver_url, deployment_name, payload), media_type="application/octet-stream")

===============================0
output:
INFO:     <IP_ADDRESS>:37382 - "POST /sessions/create HTTP/1.1" 404 Not Found
ERROR:    Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
result = await app(  # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/responses.py", line 250, in call
async with anyio.create_task_group() as task_group:
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 597, in aexit
raise exceptions[0]
File "/usr/local/lib/python3.11/site-packages/starlette/responses.py", line 253, in wrap
await func()
File "/usr/local/lib/python3.11/site-packages/starlette/responses.py", line 242, in stream_response
async for chunk in self.body_iterator:
File "/maarifa/maarifa-api/routes/workflow.py", line 451, in stream_events
session = await client.core.sessions.create()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_deploy/client/models/core.py", line 162, in create
return await self._create()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_deploy/client/models/core.py", line 167, in _create
response = await self.client.request("POST", create_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/llama_deploy/client/base.py", line 30, in request
response.raise_for_status()
File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 761, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://localhost:8000/sessions/create'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

@jdelacasa gotcha, you have to also pass the control_plane_url to the client, like this:
apiserver_url = "http://<IP_ADDRESS>:4501"
control_plane_url = "http://<IP_ADDRESS>:8000"

client = llama_deploy.Client(api_server_url=apiserver_url, control_plane_url=control_plane_url)

Thanks, initially I also got a connection error,
I realized that the control_plane was also running at http:<IP_ADDRESS>:8000 by default and I had to put in my deployment.yml file:
control-plane:
port: 8000
host: "<IP_ADDRESS>"
ahora parece que funciona.


Trll 30 - Patching the Acceptance platform

https://sphereon.atlassian.net/wiki/spaces/DEV/pages/173342725/Triall+Deployment+Patching
https://sphereon.atlassian.net/wiki/spaces/DEV/pages/173899780/TRLL-30+patch

Hey guys,
Can you please remove me from this PR ? Still hangs under my name
Thx

hi Christophe,
don't you have an unsubscribe button in the email?
I see you listed as participant, but no button to remove you. Maybe because you are the one that opened this PR.
[Image]
Get Outlook for Androidhttps://aka.ms/ghei36

From: Christophe Vico @.>
Sent: Tuesday, March 16, 2021 1:03:18 PM
To: Sphereon/acs-deployment @.>
Cc: Sander Postma @.>; Assign @.>
Subject: Re: [Sphereon/acs-deployment] Trll 30 - Patching the Acceptance platform (#2)
Hey guys,
Can you please remove me from this PR ? Still hangs under my name
Thx
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://github.com/Sphereon/acs-deployment/pull/2#issuecomment-800202389, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGNBRKXKGNEXXWQVEMTF6I3TD5CINANCNFSM4KUUIPHQ.

Indeed.  Ownership can't be transferred.
It comes to : Do you want me to Merge it or to Close it ?


E11000 duplicate key error collection: test.users unique true

I get this error when I am trying to create new user. for now the workaround is deleting the upvoteurl, downvoteurl and submittedurl index in mongodb atlas.
why it happens ?
I think because of unique constraint in schema.js for user ( submittedurl, upvoteurl and downvoteurl) has unique: true. so when we create new user by default all of them are empty/null , thus mongodb gives error about duplicate key

I understand that submittedUrls, upvotedUrls and downvotedUrls are arrays of URLs that the user has interacted it. The idea is to not have duplicates in that array (e.g.: I can only upvote URL X one time).
Upon further reading I noticed that the unique: true attribute is assign to the whole field (not the elements inside the array). Therefore eliminating the unique: true attribute might be the best solution to solve this duplicate problem.
To mantain the no-duplicates in the array, we can use the $addToSet operator when we add new values to the arrays (reference).
I fix this issue in commit https://github.com/State-Channel-4/Channel4-pocv2/commit/b743378b454347a2556da11dc382bc6ae55555c6 that will be merged in PR https://github.com/State-Channel-4/Channel4-pocv2/pull/25


Cinema Ticketing System

Project Name
Name of your project
Project Descriptione
Description of your project (What it is, what it does, how it works etc...)
Project Difficulty
Beginner, Intermediate, Advanced
Project User Stories
List of project requirements/tasks to be completed

[ ] Task 1
[ ] Task 2
[ ] Task 3
[ ] ...

Useful links and resources (Optional)
Links of useful resources related to the project
Example Projects (Optional)
Links to existing projects found around the web.

I had this project idea to add to the list but I saw you already opened an issue for it but without any details, so I thought I would try to complete the Project description

Project Name
Cinema Ticketing System
Project Description
Every cinema now has the option to buy or reserve tickets for films in advance.
In this project, you will try to build a system where users can select films and buy tickets for their favourite seats in the cinema.
Try to build a page that presents the films well and where users have an easy flow to buy the tickets. Once you are done with the user stories, why not try to add the bonus features.
Project Difficulty
Intermediate
Project User Stories - List of tasks to be completed

[ ] User can see a list of all movies currently playing in the cinema
[ ] User can select a movie and a time of the showing
[ ] User can see an overview of the cinema hall of at least 4 rows x 6 seats
[ ] User can select up to 6 seats to buy tickets for

Bonus Features

[ ] User can sort the movies by the date they are playing and filter by genre and age restriction
[ ] Different showings can be in different cinema halls with more/fewer rows and seats
[ ] User can add a discount code, student or senior citizen discount to all or some tickets
[ ] User gets a confirmation with the tickets by email

Useful links and resources (Optional)

The Movie Database
Sending Emails with JavaScript

Example Projects (Optional)

Cinema +
Build a Movie Seat Booking App


Error from server (InternalError): error when creating "oracle-database-operator.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

I use the process from here: https://ronekins.com/2021/11/11/getting-started-with-the-oracle-database-kubernetes-operator-part-1/ or here: https://github.com/oracle/oracle-database-operator

Verify install of Cert-Manager

# kubectl get pods --namespace cert-manager

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5f68c9c6dd-9h49r              1/1     Running   0          66s
cert-manager-cainjector-57d6fc9f7d-58fjn   1/1     Running   0          66s
cert-manager-webhook-5b7ffbdc98-gx72v      1/1     Running   0          66s

Install the operator from github

# kubectl apply -f https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml

namespace/oracle-database-operator-system created
customresourcedefinition.apiextensions.k8s.io/autonomouscontainerdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabasebackups.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabaserestores.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/cdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dataguardbrokers.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dbcssystems.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/oraclerestdataservices.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/pdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/shardingdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/singleinstancedatabases.database.oracle.com created
role.rbac.authorization.k8s.io/oracle-database-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-proxy-rolebinding created
service/oracle-database-operator-controller-manager-metrics-service created
service/oracle-database-operator-webhook-service created
mutatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-validating-webhook-configuration created
deployment.apps/oracle-database-operator-controller-manager created
Error from server (InternalError): error when creating "https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded
Error from server (InternalError): error when creating "https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

Checking the dns is running (coredns and flannel)

kubectl get pods -n kube-system

NAME                                       READY   STATUS    RESTARTS   AGE
coredns-dcb45f57-hzqbn                     1/1     Running   5          10d
coredns-dcb45f57-ndxqh                     1/1     Running   5          10d
etcd-verrazzano-api-2                      1/1     Running   5          10d
kube-apiserver-verrazzano-api-2            1/1     Running   5          10d
kube-controller-manager-verrazzano-api-2   1/1     Running   5          10d
kube-flannel-ds-dqcqk                      1/1     Running   5          10d
kube-flannel-ds-mc7vz                      1/1     Running   6          10d
kube-flannel-ds-xl4q6                      1/1     Running   7          10d
kube-proxy-2ff8c                           1/1     Running   5          10d
kube-proxy-9fjvm                           1/1     Running   5          10d
kube-proxy-m6qrj                           1/1     Running   5          10d
kube-scheduler-verrazzano-api-2            1/1     Running   5          10d

Firewall is stopped for this install

# systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2024-01-17 07:19:10 GMT; 8s ago
Docs: man:firewalld(1)
Process: 1474 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 1474 (code=exited, status=0/SUCCESS)
Jan 17 06:41:16 verrazzano-api-2 systemd[1]: Starting firewalld - dynamic firewall daemon...
Jan 17 06:41:16 verrazzano-api-2 systemd[1]: Started firewalld - dynamic firewall daemon.
Jan 17 06:41:16 verrazzano-api-2 firewalld[1474]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be removed in a future release>
Jan 17 07:19:10 verrazzano-api-2 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Jan 17 07:19:10 verrazzano-api-2 systemd[1]: firewalld.service: Succeeded.
Jan 17 07:19:10 verrazzano-api-2 systemd[1]: Stopped firewalld - dynamic firewall daemon.

egress and ingress rules

Egress: <IP_ADDRESS>/0	TCP	All	All
Ingress: <IP_ADDRESS>/24	TCP	All	All

What to check and install the operator?

Hello ishaan
Thanks for helping, here are details.
Kubernetes is installed according to the OCNE documentation.
Machines are running as VM on OCI in Frankfurt.
@.***
A single Subnet for all
@.***
And the security rules
@.***
All for VM have SE disabled
@.*** ~]#
sestatus
SELinux status:                 disabled
Linux Firewall is disabled too for all 4 VMs
@.*** ~]#
systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
Kubernetes version
@.*** ~]#
kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.10+2.el8", GitCommit:"a22a61e842e309e518df8edf27afee0caaa1cdf8", GitTreeState:"clean", BuildDate:"2023-12-18T14:40:09Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.10+2.el8", GitCommit:"a22a61e842e309e518df8edf27afee0caaa1cdf8", GitTreeState:"clean", BuildDate:"2023-12-18T14:49:13Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
@.*** ~]#
kubectl get nodes -o wide
NAME               STATUS   ROLES           AGE   VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION                      CONTAINER-RUNTIME
verrazzano-api-2   Ready    control-plane   12d   v1.26.10+2.el8   <IP_ADDRESS>            Oracle Linux Server 8.9   5.15.0-202.135.2.el8uek.x86_64      cri-o://1.26.3
verrazzano-wrk-3   Ready              12d   v1.26.10+2.el8   <IP_ADDRESS>            Oracle Linux Server 8.8   5.15.0-<IP_ADDRESS>.el8uek.x86_64   cri-o://1.26.3
verrazzano-wrk-4   Ready              12d   v1.26.10+2.el8   <IP_ADDRESS>             Oracle Linux Server 8.8   5.15.0-<IP_ADDRESS>.el8uek.x86_64   cri-o://1.26.3
Install the cer-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
configmap/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
Wiat some time (10 minutes)
kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5f68c9c6dd-ttlxl              1/1     Running   0          88s
cert-manager-cainjector-57d6fc9f7d-jncr2   1/1     Running   0          88s
cert-manager-webhook-5b7ffbdc98-qhmsd      1/1     Running   0          88s
kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5f68c9c6dd-ttlxl              1/1     Running   0          10m
cert-manager-cainjector-57d6fc9f7d-jncr2   1/1     Running   0          10m
cert-manager-webhook-5b7ffbdc98-qhmsd      1/1     Running   0          10m
Setup the database operator
kubectl apply -f https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml
namespace/oracle-database-operator-system created
customresourcedefinition.apiextensions.k8s.io/autonomouscontainerdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabasebackups.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabaserestores.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/cdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dataguardbrokers.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dbcssystems.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/oraclerestdataservices.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/pdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/shardingdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/singleinstancedatabases.database.oracle.com created
role.rbac.authorization.k8s.io/oracle-database-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-proxy-rolebinding created
service/oracle-database-operator-controller-manager-metrics-service created
service/oracle-database-operator-webhook-service created
mutatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-validating-webhook-configuration created
deployment.apps/oracle-database-operator-controller-manager created
Error from server (InternalError): error when creating https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: context deadline exceeded
Error from server (InternalError): error when creating https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: context deadline exceeded
Same error
Error from server (InternalError): error when creating https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: context deadline exceeded
Error from server (InternalError): error when creating https://raw.githubusercontent.com/oracle/oracle-database-operator/main/oracle-database-operator.yaml: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: context deadline exceeded
What do you think?
Thanks
David
De : Ishaan Desai @.>
Date : jeudi, 18 janvier 2024 à 09:32
À : oracle/oracle-database-operator @.>
Cc : DAVID ADDE @.>, Author @.>
Objet : [External] : Re: [oracle/oracle-database-operator] Error from server (InternalError): error when creating "oracle-database-operator.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://...
Hi david, you seem to have all the conditions met. Can you just retry once; also share you environment details
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/oracle/oracle-database-operator/issues/80*issuecomment-1898018773__;Iw!!ACWV5N9M2RV99hQ!KNjmV-FLjlR4jL7FThbgoCyZO9WtFD1MSPhqXVcwbjgFw3oUoJ3FkB0GQXvH0RqI_q6TJWTKJ8vctiioNdpSc41YhQ$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ALBDG5N463IBGLF6XEKB5WLYPDMYZAVCNFSM6AAAAABB6BQSAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJYGAYTQNZXGM__;!!ACWV5N9M2RV99hQ!KNjmV-FLjlR4jL7FThbgoCyZO9WtFD1MSPhqXVcwbjgFw3oUoJ3FkB0GQXvH0RqI_q6TJWTKJ8vctiioNdraxbIVfg$.
You are receiving this because you authored the thread.Message ID: @.***>

helm install of cert-manager did the trick
helm install   cert-manager jetstack/cert-manager   --namespace cert-manager   --create-namespace   --version v1.13.3 --set installCRDs=true --set webhook.hostNetwork=true --set webhook.securePort=10260


Add some comments & fix some nits

Add comments where I felt they were missing.
Change log2_pow2 to use simpler Implementation.

This is only used in the trusted setup anyway.

Move a bytes_to_kzg_proof call up, to fail faster.
Change (uint64_t)1 to 1ULL to be more concise.
Move some trusted setup functions down to its section.
Group trusted setup free funcs together.
Update error message for FIELD_ELEMENTS_PER_BLOB in header.
Replace // comments with /* */ comments.
Remove some excess parentheses.

Other than my comments above, I think it's good to go.


Replace Toolbar showHistory group with individual showRedo and showUndo

Replace Toolbar showHistory group with individual showRedo and showUndo to give more granular control of icon display
This is breaking change as showHistory has been removed

FYI this is breaking change as showHistory was removed


Added frequency cropping option with fmin and fmax to BaseTFR.crop()

Reference issue
Fixes #5868
What does this implement/fix?
Added frequency cropping functionality to BaseTFR.crop method.
Additional information
Works like:
tfr.crop(tmin=0.2, tmax=0.4, fmin=5, fmax=7)

Note that for the method, i implemented a _freq_mask() function in utils.numerics,
which is mostly similar to _time_mask() (except by the errors thrown).
Since the _freq_mask function is (for now) only needed by BaseTFR.crop(),
it could alternatively be implemented in BaseTFR.crop(), or integrated into _time_mask().

@larsoner Thanks for noticing! That should do it.

Travis is complaining

Travis is complaining

Nah Travis was green on the latest commit. Though now of course there is a merge conflict, because nothing should be easy :)

Merge conflict is resolved.

@mmagnuski feel free to merge once you are happy

let's not maintain code that we don't use. +1 to remove option to pass
sfreq=None

sfreq=None default in _freq_mask removed

ok. Can you just do:
if tmin is not None or tmax is not None:
and
if fmin is not None or fmax is not None:
with the slice() then I promise I stop.

also @DiGyt I see that you opened a PR from your master branch. You should not do this. Currently I cannot push to your PR to help as it may create a mess with your own master branch that should be in sync with mne-python upstream master

also @DiGyt I see that you opened a PR from your master branch. You should not do this. Currently I cannot push to your PR to help as it may create a mess with your own master branch that should be in sync with mne-python upstream master

@agramfort Yes! I already found it to be inconvenient too. Won't happen again.

Great, seems that everyone is happy now, right?

thx a lot @DiGyt

That's probably better than what I did in #6104

@larsoner - You are referring to "always slice" possiblility? That would be nice, but to create the slices we would have to use np.where anyway, as numpy does not have efficient find n first or find n last functions.


Check why unit_d is not reflecting in France

Mock html has defined 4 units for France

Parsed json only lists 3 for France

This looks to be expected behaviour due to this issue
@IanMayo, just wanted to confirm if this indeed valid behaviour.

Yes - we should eventually include unit_d, once we handle the rowspan content.

Okay. Is it just going to be the Power and RR columns that may be merged, or should be check the same for other columns as well?

I didn't see a pattern, so it appears any column could be merged.  Aah, except for the first one - that's the unique identifier.

@IanMayo, also check if the tables have colspan contents in the data section. I see one possibility where they might merge multiple columns to indicate non-availability or non-applicability (N.A.)
If there're such rows, then we'll need to handle them as well.

I had removed the Too few properties exception. I believe we should retain it to make the processing sturdier.


Allow using multiple files for single POJO.

Currently single POJO is mapped with single flat file. It is a good feature to support multiple file formats for single POJO.
User can filter for required file format in parse method.

This feature does not seems to have any ROI to put in the efforts. Closing the ticket.


Expand/Experiment laserpanel expansion

Add in laserpanel raw-save function

myfile.zip
Draw circle. Click Save in LaserPanel. Get file.
Document type : LHYMICRO-GL file
File version: 1.0.01
Copyright: Unknown
Creator-Software: MeerK40t v0.8.0023 Beta15 git

%0%0%0%0%
IBzzzzzzzzzzzzzzzzzzzzz157Rzzzzzzzzzz206S1P
ICV2551031128000000CNTRS1E|mMaRwDaUdMaRsMaRoMaRnMaReDaUeMaRjMaRjMaRiMaRhMaRhMaRfDaUaMaRgMaRfMaRgMaRfMaRfMaRfMaReMaRfMaRbDaUbMaReMaReMaReMaRdMaReMaRdMaReMaRdMaRdMaRdMaRdDMaURdMaRdMaRdMaRcMaRdMaRcMaRdMaRcMaRdMaRcMaRcMaRcMaRdMaRbDaUMaRcMaRcMaRcMaRcMaRbMaRcMaRcMaRcMaRcMaRbMaRcMaRbMaRc...R|cMaRdDaU|hFNSE-

This file is fairly close to being compatible with Whisperer and the various chinese software. It however has some altered/upgraded code structures. For example the \n commands means basically go ahead and send that. And the - command triggers the wait-finished (without that it will automatically try waiting finish without ever having sent an F command before the NSE. So you could do simple rapid commands without requiring the code be entirely within programmed mode.

The raw-save only works for M2 Nano.
I've hardcoded the file type into the panel rather than defer to the device driver.


Multiple page version

Pandocs is currently being served as a one long page, following the spirit of the original document.
Some users (as #289 ) expressed the desire to be able to navigate the document split in different pages (one per chapter?). It should be trivially possiblle to add another HTML export target having one chapter / subchapter per page, while also keeping the current version.

Closed by #292


Add HTTP CONNECT Handler

Hijacks request connection and sends it to the next handler

Thanks! This looks cool. Do you have an example of how this would be used?
And does this work with the current proxy module? Should it, I wonder?

@kkroo Any interest in having this merged?


Update FAQ #quarantine_measures: Replace outdated info

#quarantine_measures

Internal Tracking ID: EXPOSUREAPP-13527

Also https://www.coronawarn.app/en/faq/results/#voluntary_self_quarantine needs to be reviewed, which states "A minimum of 7 days, preferably 10 days, is recommended for the period of self-quarantine from the date of the last increased risk encounter. "

Good morning @MikeMcC399,  Thanks for your comment, we will update the article you mentioned in a separate PR.


Not able to add kubernetes metadata for logs collected from additional log files from kubernetes nodes

Discussed in https://github.com/signalfx/splunk-otel-collector-chart/discussions/1345

Originally posted by ankurnema July 18, 2024
Hello,
We have a usecase where we are trying to tail log files on kubernetes nodes and have configured configuration under below
extraFileLogs: {}

For our usecase we are trying to tail log files in kubernetes empty directory from below location
/var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/test/test.log

In above * usually is pod uid.
Now I want to enrich logs collected with kubernetes metadata by using k8sattributes processor. How can I do this?
As per Code https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/templates/config/_otel-agent.tpl#L743C7-L752C19 there is no option to add any additonal processor.

Please open a support case and we will be happy to provide assistance. See https://splunk.my.site.com/customer/s/need-help/create-case to get started. Thanks!


CallbackReturn should not be a type

Instead it's values should be c_int constants.
This is a thing because you indeed can return arbitrary values from the callback, for example the ACTION callback of IupText is allowed to return a character value.

OK, feel free to make the change and I can bump the version and publish or you can if you want to be a co-owner of the package.


Set gain register not implemented

As in the AD7173, the gain value can be set using set gain registers. It is not implemented in the code

@tharaka27  It is implemented at line 279 of  file

Thank you. I'll close the issue


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-main" workflow

Codecov Report

Merging #787 (3a88df5) into main (b87052f) will not change coverage.
The diff coverage is n/a.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@           Coverage Diff           @@
##             main     #787   +/-   ##
=======================================
  Coverage   95.69%   95.69%
=======================================
  Files           1        1
  Lines         395      395
  Branches       53       53
=======================================
  Hits          378      378
  Misses         17       17


Component to create: textfield

Official link to component description & demo: http://dev.office.com/fabric/components/textfield
Screenshot of web analytics for the original http://officeuifabric.com site by @andrewconnell. Use this to see what are the most popular components: http://imgur.com/gallery/buUcyQv

This has currently been implemented & submitted by @rolandoldengarm, but needs some things to be addressed...

[ ] converted from an attribute => element directive
[ ] add support for multiline
[ ] add support for placeholder
[ ] add support for underline
[ ] add more robust docs to directive


Roll Engine from 4bf9d638c219 to ab01b042c2cd (1 revision)

https://github.com/flutter/engine/compare/4bf9d638c219...ab01b042c2cd
2021-06-25<EMAIL_ADDRESS>Roll Skia from 9c81a4b7047d to 0b04b6b16f1f (1 revision) (flutter/engine#26957)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Separate pools for reading and writing to SQL database

I'm working on an spring-boot web application application using spring-boot-jpa (Spring Data JPA) to store and retrieve data in a relational database.
I have a brief idea about HikariCP and using it to manage multiple connection pools. But, I want to perform read operations (DQL) and write operations (DML) on the same database using separate connection pools respectively.
Is there a way to achieve the same??

This is primarily a Spring Data question.  HikariCP has not limitation that would prevent you from creating two pools in one application pointing to the same DB.  It's no different than many application servers connected to the same DB.  I suggesting asking on stackoverflow.com or over on the Spring Data forums.


(PDB-4236) blacklist jackson-databind permanently

We don't use it in puppetdb, and we don't want to constantly update
it due to CVEs

Test PASSed

CLA signed by all contributors.


400ms backtrack

I'm very excited about adding 400ms backtrack to Osiris. I used #1922 before but it's outdated already. Would you mind adding 400ms backtrack to Osiris? @danielkrupinski

i have it in my paste and it still works, the config stuff is in Backtrack.cpp now tho

i have it in my paste and it still works, the config stuff is in Backtrack.cpp now tho

Can you help me please? wirekurosastak#3796

how do i get this in my paste


fixed path for wdio.cmd

It is an error in the windows environment.
$ npm test

><EMAIL_ADDRESS>test d:\home\gulp-webdriver
> gulp webdriver

[16:46:36] Using gulpfile d:\home\gulp-webdriver\gulpfile.js
[16:46:36] Starting 'webdriver'...
module.js:339
    throw err;
    ^

Error: Cannot find module 'webdriverio\bin\wdio.cmd'
    at Function.Module._resolveFilename (module.js:337:15)

Fixed path for wdio.cmd.
webdriverio/bin -> webdriverio/../.bin (node_modules/.bin)

node_modules/.bin
  wdio          // shell
  wdio.cmd  // windows bat file

webdriverio/bin
  wdio         // node.js file

Hi,
I ran into this issue as well. Problem is, this depends on your version of npm. On npm 2, which ships with node 4, (current LTS), the path will work as it is currently resolved.
On npm 3 however, shipping with node 5, the new behavior is to flatten the dependency tree, causing the .cmd to go in a top-level .bin.
Means there is probably a check to be made to correctly resolve the dependency according to the environment, though this feels a bit hacky :/

This has already been fixed.


feat(validateMessage): Allow semver commits

In this PR the IGNORED regular expression is updated to skip commits that contain just a valid semver as the ones generated by npm version.
Closes #13

Current coverage is 96.22%

Merging #17 into master will increase coverage by +0.07% as of 5a17fa8

@@            master     #17   diff @@
======================================
  Files            1       1
  Stmts           52      53     +1
  Branches         0       0
  Methods          0       0
======================================
+ Hit             50      51     +1
  Partial          0       0
  Missed           2       2

Review entire Coverage Diff as of 5a17fa8

Powered by Codecov. Updated on successful CI builds.

I need to give you an apology. For some reason, I want watching this repository so I want getting notifications about issues and pull requests. I am so incredibly sorry. Please forgive me. I really feel awful. I hope my unresponsiveness doesn't hinder you contributing more in the future.
As for your specific pull request, I'll give it a closer look soon.

This is great. Thanks for the work. Unfortunately it no longer merges cleanly (sorry about that!) If you could please rebase and update the PR to the current codebase, I'd really appreciate it. Thank you.

@kentcdodds Rebased

Excellent. Thanks!


Cloning Apps ASE v1 to v2

Can we use the cloning feature to migrate Apps from ASE v1 to ASE v2? Thanks a lot for the info. Regards.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: daf14087-1765-4636-805c-6faec03d8a9b
Version Independent ID: 4a6b910d-403e-997a-b010-0942e8ef624c
Content: Clone app with PowerShell - Azure App Service
Content Source: articles/app-service/app-service-web-app-cloning.md
Service: app-service-web
GitHub Login: @ahmedelnably
Microsoft Alias: aelnably

@michelnubiral Please help with the link of document on which above document feedback request is created.
In case your query in not related to document content, open a new forum thread in MSDN forum or Stack Overflow as it could be better place to get help on your scenario. These forum community  members could provide their expert guidance on your scenario based on their experience. Thanks.

Hi @MohitGargMSFT thank you for your assist. This is the link:
https://docs.microsoft.com/en-us/azure/app-service/app-service-web-app-cloning
Thank you a lot for your prompt response.

@michelnubiral Yes, as long as the tier is standard or higher, you should be able to clone the app, even between ASEs. Please note this will only copy the site settings and not other ASE configurational settings that you may need to manually set.


Copy button - not working for ip address

Environment

Nautobot version (Docker tag too if applicable): 1.4.1
Python version: 3.7.13
(tested also on demo nautobot)

Steps to Reproduce

Open any device object with IP address assigned in Nautobot GUI
Click on a "copy" button next to Primary IP address when hovering with mouse

Expected Behavior
The IP value should have been copied in clipboard.
Observed Behavior
Nothing is copied (if a clipboard contained something before, it stays there without any change).
Probably the issues is due to fact that the IP address HTML element is missing the ID ipv4 that is supposed to be used for element selection (see image) because the "copy" button works for device name though (in title) where element ID is properly defined.
btw. I didn't want to create two seperated issues for this topic / for another (un)expected behavior but are you aware that when the copy buton is used for a device name it copies tha device name + it adds a whitespace in front of and behind the name itself? The clipboard is NOT just the name itself (e.g. ams01-dist-01)
I would be nice to fix this and also extend in the future with already existing request: https://github.com/nautobot/nautobot/issues/1519 (just saying :-) )

Thanks for the report! Looks like this was broken by #1833.


V1 dashboard with live data

Dashboard with live data and detail drawer. Note some TODOs left once API payloads are fleshed out.
https://user-images.githubusercontent.com/17093033/155274032-08f706f7-ff36-4a12-b91f-9c00e2e48e0b.mov

There's some minor padding changes needed for the id within the highlighted row & a border, see the mockup:

what it looks like now:


fzfp error: unknown option.

Thank you for the scripts! :)
I get an error running the popup though:
$ ls --color=always | fzfp --ansi --width=50% --height=50%
unknown option: --width=50%

Tmux version: 3.1c
Thanks again!

Please upgrade tmux to 3.2 later.

popup feature hasn't released in a stable tmux version.

Please upgrade tmux to 3.2 later.

popup feature hasn't released in a stable tmux version.

I got the same error (I guess?)

$ ls | fzfp --ansi --width=50% --height=50%
awk: syntax error at source line 1
 context is
        {match($0, >>>  /[0-9]+\.[0-9]+/, <<<
awk: illegal statement at source line 1
awk: illegal statement at source line 1
unknown option: --width=50%

Here is my tmux version:
$ tmux -V
tmux 3.3a

I had the same issue for a long time. I didn't know why it worked sometimes and sometimes it didn't.
Realised that it was working on RHEL-based servers but not on Debian-based.
Then I looked at the awk version:
% awk --version
mawk 1.3.4 20230525
Copyright 2008-2022,2023, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

random-funcs:       arc4random_stir/arc4random
regex-funcs:        internal

compiled limits:
sprintf buffer      8192
maximum-integer<PHONE_NUMBER>854775808

fzfp doesn't seem to work with mawk. You need to install GNU awk, gawk. awk gets aliased to gawk and overrides mawk when it's installed.


save() output advice: usepgfplotslibrary

save() function advises:
Please add the following lines to your LaTeX preamble:
\usepackage[utf8]{inputenc}
\usepackage{fontspec} % This line only for XeLaTeX and LuaLaTeX
\usepackage{pgfplots}
\usepgfplotslibrary{plotmarks}

However, pdfLaTeX casts an error:
I can't find file `pgflibrarypgfplots.plotmarks.code.tex'. \usepgfplotslibrary{plotmarks}
Emergency stop. \usepgfplotslibrary{plotmarks}

But with \usetikzlibrary{plotmarks} it works fine.
Should it be updated?

Fixed via 0a97368fd44df3bb988fca0d51cd9301ea8766bd.

Thanks for fixing.
How can I disable or suppress this output?


[folding] instead of  '...' show body content for short methods

This function will allow to toggle the code view to "compact" mode. Compact mode means that short methods will be displayed as a one-liner. It looks similar to a folded methods, but the method's body remains visible. A method is conisdered short, if it fits on one line in the current viewport, or if it is shorter than a configurable number of lines / characters.
Technically a compacted method simply replaces all line-breaks with a blank/space, while it has no focus.
The feature operates completely in memory, the underlying editied file remains unchanged. If a compacted method is focused, by clicking or moving the cursor inside its body, it transforms to its original view, so that editing looks and works as expected. The cursor simply remains at the clicked position, while the line-breaks are restored. By Clicking outside the method being edited, it returns to compact mode.
Benefit is better readability for short methods. Compared to folding, all code remains visible, and no toggles are required. Optimizes real estate use on wide screen displays.
Example for "Normal View"
PHP:
public function Test($aParam)
{
    $this->Test = $aParam;
    return $this;
}

CSS:
body {
    background:#f00;
    color:#000;
    border:1px solid #ff0000;
    font-size:1.6rem;
}

"Compact View":
PHP:
public function Test($aParam) {$this->Test = $aParam; return $this;}
CSS:
body {background:#f00; color:#000; border:1px solid #ff0000; font-size:1.6rem;}

Nice idea!

it should be added, that in the case of LESS, recursion is neccessarry:
.test {
    background:#f00;
    color:#0f0;
    img {
        width:100%;
        boder:1px solid #f00;
    }
}

would transform to:
.test {background:#f00; color:#0f0;
    img {width:100%; boder:1px solid #f00;}
}

Focusing ".test" class by setting the cursor on one of its attributes would leave "img" definition compacted.

dup of #70794


Architecture diagram

Hi,
What changes I need to make to the diagram in page 4 of the paper "
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks" to show that Elmo is concatenated?

I think it would be okay to just mention it in the diagram or in the paper. "Concatenated ELMO embeddings" instead of e.g. Word Emb.

And what would be the dimension after concatenating, if word embeddings are  200 d?

ELMo embeddings are 1024d per layer, 3 layers => 3072 dimensions for concatenated ELMo embeddings


Match the node start API to the cluster start API

W/ regards to wait-other-notice
@mambocab and @kishkaru to review. Running CI here: http://cassci.datastax.com/view/Dev/view/ptnapoleon/job/ptnapoleon-cassandra-2.2-dtest/
Should not affect DSE

:+1:

I have not run the changes to the CLI, but +1 from me.


ContribClass instance in registry and generalize commands

This PR:

creates an instance of the ContribClass from ContribClass.import_call and saves it in registry
calls methods of instance, like ContribInstance.train(*args, **kwargs)
For Comet, it uses a local config file to pass experiment_key down the pipeline

TODO: need to call contrib_command("METHOD_NAME") for places to add hooks.

I was thinking that this system might be a good way to add support for Jupyter notebooks (mentioned in the issues) and other environments too. Anything outside the main focus of the ludwig pipeline could be added through the contrib classes and methods.

This looks really nice. We can figure out together what additional functions need to be added and where.
Now we have a "import" time function and a "cli command" time function, we can figure out a bunch of others, like "visualization show" or "before training", "end of epoch", "end of training", "before prediction" and "after prediction".
One really minor consideration, which at the moment doesn't make any difference: at the moment we have one single registry for classes and instances, so each time we go over the registry and check if an instance has been created ( https://github.com/uber/ludwig/blob/89157e7e8cb9b5cdd3ac4fc7c618d0f0c1e810b7/ludwig/contrib.py#L117-L118 ). This is fine for now that there's only one contrib class. But I can imagine a moment when there are 10 contrib classes, of which only one or 2 are initialized at a time. Maybe keeping a separate list / set of initialized instances to iterate over would be a better option then.
Finally, do you want me to merge this already or should we figure out the visualization part first? (consider the viz part of the code is the one that received less love in terms of polish so far in the codebase)

Thanks! I think this will work nicely for many uses (I hope!)
Yes, you are right... that should be a faster method of finding the activated contribs. (contribs should probably be a folder too, as each one of these classes can get big).
I'd say go ahead and merge it, and we can continue to refine, if that is fine with you. And we can get more eyes on it.

I made a followup PR #277 that addresses three items.


PowerShell file hash differs when it is multi-line.

Created a package that was a PowerShell script, multi-line. When Gorilla downloaded the file, the hash was different than when I added it to the catalog, so it threw an error and exited.
Pretty sure it is because of Windows end-line characters. When I re-created the script and put all the lines on one line separated by a semicolon it worked flawlessly. It may have something to do with how Gorilla gets this script from the web server (straight up downloading it, or "re-creating" the file in real-time). Either way, this seems problematic for larger PowerShell scripts.

You should make sure that when you save the files, your IDE isn't adding extra stuff. Can't say I've seen this error and we have several large powershell scripts.
It could be some kind of carrier return or encoding issue.
We use git with an automation pipeline - are you manually uploading these files yourself?

A hash should not be affected by line-endings.
How are you hosting the files? If you manually download the files from the server and check the hash, what do you get?

The file was created on VS Code on a Windows machine, using Windows line endings. We re-typed the script (multi-lined), uploaded it, and manually kicked off a Gorilla run on the endpoint, and it still returned a different hash on the endpoint than what was originally created and being uploaded onto the webserver.
I'll look more into what was going on; one of my colleagues seems to think it has something to do with the process of getting the files onto the webserver (committing to a git repo, then being pulled over via Chef).

We now suspect that the culprit is haproxy changing the file to LF end-of-line upon a request for the file from the web server. We're going to attempt re-creating the file with LF end-of-line sequences, re-uploading to the server, and then trying a gorilla run again.
Apparently EOL sequences do change the hash. /shrug

It's because it's a different encoding.

Glad you found a potential cause!
Ideally your web server would serve content exactly as it is, without any modifications. But understanding what is really going on allows you to work around it.
Any change to the file (content, line endings, or encoding, etc) would change the hash.
Let us know the outcome of testing the line ending changes.

@YourCrush @gyamada619 Any luck getting confirmation this was caused by haproxy?

As far as we can tell, yup, that was it. Sorry for not getting back over here sooner to confirm! Appreciate you checking in and verifying.


Error on backup nginx-example app with PV (AWS GP2)

When I try to backup the nginx example app, it fails saying that the volume does not exists.
Command executed:
 velero backup create nginx-backup2 --selector app=nginx
Nginx manifest
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-example
  labels:
    app: nginx

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-logs
  namespace: nginx-example
  labels:
    app: nginx
spec:
  storageClassName: gp2
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: nginx-example
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        pre.hook.backup.velero.io/container: fsfreeze
        pre.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--freeze", "/var/log/nginx"]'
        post.hook.backup.velero.io/container: fsfreeze
        post.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--unfreeze", "/var/log/nginx"]'
    spec:
      volumes:
      - name: nginx-logs
        persistentVolumeClaim:
          claimName: nginx-logs
      containers:
      - image: nginx:1.7.9
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/var/log/nginx"
          name: nginx-logs
          readOnly: false
      - image: gcr.io/heptio-images/fsfreeze-pause:v0.11.0
        name: fsfreeze
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: "/var/log/nginx"
          name: nginx-logs
          readOnly: false

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: my-nginx
  namespace: nginx-example
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Error
time="2019-03-12T18:38:13Z" level=info msg="Backup completed with errors: error getting volume info: rpc error: code = Unknown desc = InvalidVolume.NotFound: The volume 'vol-0b3d2XXXX' does not exist.\n\tstatus code: 400, request id: 2e937281-c6a7-4063-a704-" backup=velero/nginx-backup2 logSource="pkg/backup/backup.go:297"

Velero:
Client:
	Version: v0.11.0
	Git commit: ab2fc65c02ecdcb9bb1690e3afe67a3b3196aebc
Server:
	Version: v0.11.0

@fcappi this often indicates a permissions issue. Can you double-check that you have the IAM policy correctly set up?

Hi,
I added the required policies(https://github.com/heptio/velero/blob/master/docs/aws-config.md#create-iam-user) to my k8 cluster nodes using kops additional policies (https://github.com/kubernetes/kops/blob/master/docs/iam_roles.md#adding-additional-policies).
Velero was able to create files, so that means the policy is working fine, but for some reason, is not being able to retrieve the volumes.
kops cluster spec
apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2019-03-13T17:08:11Z
  name: k8-cluster
spec:
  additionalPolicies:
    master: |
      [
        {
        "Effect": "Allow",
        "Action": [
          "autoscaling:DescribeAutoScalingGroups",
          "autoscaling:DescribeAutoScalingInstances",
          "autoscaling:SetDesiredCapacity",
          "autoscaling:DescribeLaunchConfigurations",
          "autoscaling:DescribeTags",
          "autoscaling:TerminateInstanceInAutoScalingGroup"
        ],
        "Resource": ["*"]
        }
      ]
    node: |
      [
          {
              "Effect": "Allow",
              "Action": ["route53:ChangeResourceRecordSets"],
              "Resource": ["arn:aws:route53:::hostedzone/*"]
          },
          {
              "Effect": "Allow",
              "Action": ["route53:ListHostedZones", "route53:ListResourceRecordSets"],
              "Resource": ["*"]
          },
          {
           "Effect": "Allow",
           "Action": [
               "ec2:DescribeVolumes",
               "ec2:DescribeSnapshots",
               "ec2:CreateTags",
               "ec2:CreateVolume",
               "ec2:CreateSnapshot",
               "ec2:DeleteSnapshot"
           ],
           "Resource": "*"
          },
          {
           "Effect": "Allow",
           "Action": [
               "s3:GetObject",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:AbortMultipartUpload",
               "s3:ListMultipartUploadParts"
           ],
           "Resource": [
               "arn:aws:s3:::S3-Bucket/*"
           ]
          },
          {
           "Effect": "Allow",
           "Action": [
               "s3:ListBucket"
           ],
           "Resource": [
               "arn:aws:s3:::S3-bucket"
           ]
          }
      ]

Hmm. is the master schedulable? Is it possible the velero pod is running on the master (which doesn't look like it has the policy)?
If not - can you confirm that the volume with that ID does actually exist?

@skriss The pod is running on worker nodes. Even after adding the policy to master, the issue is the same.
Volume can be retrieved from AWS console and AWS Cli.

I think maybe it's trying to get the volume from a different region. How Velero define the region it should use to get the volume from?

You specify it in the VolumeSnapshotLocation, under .config.region (see https://heptio.github.io/velero/v0.11.0/api-types/volumesnapshotlocation.html#aws)

So that was the issue. I was not specifying the region on  VolumeSnapshotLocation. Now I was able to backup and restore without any problems.
Thanks @skriss