MOBILES
LAPTOPS
ACCESSORIES
ABOUT US
CONTACT US
❮
❯
MOBILES
TOP SELLS
Realme 8 5G
AMD Ryzen 3 Dual Core Processor
4 GB RAM | 128 GB ROM
6.5 inch Full HD+ Display
48MP + 2MP + 2MP | 16MP Front Camera
5000 mAh Battery
₹14,999
Buy Now
APPLE iPhone XR
64 GB ROM
A12 Bionic Chip Processor
6.1 inch Display
12MP Rear Camera | 7MP Front Camera
Brand Warranty of 1 Year
₹27,999
Buy Now
OPPO Find X
8 GB RAM | 256 GB ROM
6.4 inch Full HD Display
16MP + 20MP | 25MP Front Camera
3730 mAh Li-polymer Battery
Snapdragon 845 Processor
₹19,999
Buy Now
POCO F1
6 GB RAM | 64 GB ROM
6.18 inch Full HD+ Display
12MP + 5MP | 20MP Front Camera
4000 mAh Li-polymer Battery
Qualcomm Snapdragon 845 Processor
₹12,999
Buy Now
LAPTOPS
TOP SELLS
MSI Modern 14
Intel Core i5 Processor (10th Gen)
8 GB DDR4 RAM & 512 GB SSD
64 bit Windows 10 Operating System
35.56 cm (14 inch) Display
1 Year Carry In Warranty
₹42,000
Buy Now
Mi Notebook Horizon
Intel Core i7 Processor (10th Gen)
8 GB DDR4 RAM & 512 GB SSD
64 bit Windows 10 Operating System
35.56 cm (14 inch) Display
1 Year Manufacturer Warranty
₹45,000
Buy Now
DELL Inspiron Core
AIntel Core i5 Processor (11th Gen)
8 GB DDR4 RAM & 512 GB SSD
64 bit Windows 10 Operating System
35.56 cm (14 Inch) Touchscreen Display
1 Year Onsite Warranty
₹45,999
Buy Now
APPLE MacBook Air M1
li>Apple M1 Processor
8 GB DDR4 RAM & 256 GB SSD
Mac OS Operating System
33.78 cm (13.3 inch) Display
1 Year Limited Warranty
₹72,000
Buy Now
ACCESSORIES
TOP SELLS
Mi Smart Band 5
touch AMOLED Display
AMOLED Display
Sports Mode
Battery Life: Up to 14 days
Water Resistant
₹1,799
Buy Now
OnePlus Buds
Bluetooth version: 5
10 hours playtime for 10 mins of charge.
30 hours of full Playtime
Dolby Atmos
IPX4 resistant
₹1,200
Buy Now
SanDisk OTG Drive
Lightning, USB 3.0|128 GB
Metal, Plastic
For Desktop Computer, Mobile
Color:Silver, Black
₹799
Buy Now
Realme Power Bank
Capacity: 20000 mAh
Lithium Polymer Battery
Power Source: AC Adapter
Micro, Type-C Connector
Charging Cable Included
₹999
Buy Now


PLANET SICK-BOY: http://www.sick-boy.com "We Put the SIN in Cinema"
Films based on true stories are unique cinematic animals. To begin with, if the story is popular enough, viewers already know the ending (like Titanic). Even if the tale isn't particularly well-known, studios can ruin a film's finale by having the main character appear on the talk show circuit prior to release, so potential viewers will know he/she doesn't die no matter how much adversity they face in the film (like The Hurricane). And if you've got a problem with the film's story being boring, can you really complain if it's based on actual events? I felt that way about The Perfect Storm and was besieged with e-mail from people accusing me of not feeling sorry enough for the characters – like my review was really damning the crew of the Andrea Gail and their families instead of the film.
Men of Honor was cobbled together with the same cut-rate glue they used to assemble The Perfect Storm. While its central character isn't quite as unlikable as the money-grubbers from the Andrea Gail, the film offers precious little besides a formulaic story, phoned-in performances and button-pushing racial issues calculated to cause white viewers to hang their heads in shame as they drive their SUVs back to the suburbs.
Honor is about Carl Brashear, who is portrayed here by Cuba Gooding Jr. (Chill Factor). The film follows Brashear's life from his family's Sonora, Kentucky dirt farm to a military career during which he became the first black deep-sea diver in U.S. Navy history. Of course, the journey from Point A to Point B is full of adversity, mostly at the hands of the racism Brashear encountered in the ‘60s Navy. He starts his career as a cook on the seas of the South Pacific, of all the horrible things.
After finally clawing his way through two years of training in casseroles and soufflés, Brashear lands at a New Jersey dive and salvage school, which is run by redneck Master Chief Billy Sunday (Robert De Niro, Meet the Parents). Sunday is a drunk with a corncob pipe (but, sadly, no button nose) and has recently been assigned this training position after severely damaging his lungs during a daring dive in the Mediterranean. He immediately takes a dislike to Brashear's hue and makes him work 10 times harder than anybody else in the training program.
The rest of Honor is pretty predictable (aside from a startling and grisly injury to one of the characters). We've already seen countless films where two chest-thumping Alpha males butt heads, and it's not any different here just because it's based on a true story. Honor is slowed down by obligatory romantic threads involving Brashear's Harlem librarian (Aunjanue Ellis, In Too Deep) and Sunday's strange relationship with a beautiful woman played by Charlize Theron (who is making her third feature film appearance in less than a month - The Yards and Bagger Vance).
Honor was the writing debut of Scott Marshall Smith and was directed by George Tillman Jr. (Soul Food). The film features Gooding's best performance since Jerry Maguire, but that's not really saying much if you've seen what he's done lately. The best and most exciting scene is given away in the trailer. It fails to inspire in nearly every way, but does happen to be the second boat film this year to feature David Keith (U-571).
1:20 – R for adult language and graphic violence
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Files
File Description
~/.ssh/ Directory for user-specific SSH configuration
~/.ssh/authorized_keys Lists public keys authorized for logging into this user
~/.ssh/config Per-user config file. Can specify how to connect, with which keys etc
~/.ssh/id_* Key files, both public and private
~/.ssh/known_hosts Contains list of public host keys known to user
/etc/ssh/ssh_config Global SSH client configuration
/etc/ssh/sshd_config SSH server configuration


Versions:
HypixelStats ,
Node ,
Chrome ,
and Electron .
Players:
Do /who
again if not everyone shows up. You can report players by doing /msg _report PlayerName
in hypixel.


Recipe 12.2. Counting Tags in a DocumentCredit: Paul Prescod ProblemYou want to get a sense of how often particular elements occur in an XML document, and the relevant counts must be extracted rapidly. SolutionYou can subclass SAX's ContentHandler to make your own specialized classes for any kind of task, including the collection of such statistics: from xml.sax.handler import ContentHandler import xml.sax class countHandler(ContentHandler): def _ _init_ _(self): self.tags={ } def startElement(self, name, attr): self.tags[name] = 1 + self.tags.get(name, 0) parser = xml.sax.make_parser( ) handler = countHandler( ) parser.setContentHandler(handler) parser.parse("test.xml") tags = handler.tags.keys( ) tags.sort( ) for tag in tags: print tag, handler.tags[tag] DiscussionWhen I start working with a new XML content set, I like to get a sense of which elements are in it and how often they occur. For this purpose, I use several small variants of this recipe. I could also collect attributes just as easily, as you can see, since attributes are also passed to the startElement method that I'm overriding. If you add a stack, you can also keep track of which elements occur within other elements (for this, of course, you also have to override the endElement method so you can pop the stack). This recipe also works well as a simple example of a SAX application, usable as the basis for any SAX application. Alternatives to SAX include pulldom and minidom. For any simple processing (including this example), these alternatives would be overkill, particularly if the document you are processing is very large. DOM approaches are generally justified only when you need to perform complicated editing and alteration on an XML document, when the document itself is made complicated by references that go back and forth inside it, or when you need to correlate (i.e., compare) multiple documents. ContentHandler subclasses offer many other options, and the online Python documentation does a pretty good job of explaining them. This recipe's countHandler class overrides ContentHandler's startElement method, which the parser calls at the start of each element, passing as arguments the element's tag name as a Unicode string and the collection of attributes. Our override of this method counts the number of times each tag name occurs. In the end, we extract the dictionary used for counting and emit it (in alphabetical order, which we easily obtain by sorting the keys). See AlsoRecipe 12.3 for other uses of SAX. |


Grid Based Design
Grid based design is a way of laying out a web page using a grid structure, made up of rows and columns. For responsive design, the most common grid layout utilises 12 columns, and a total width of 100%, and will shrink and expand as you resize the browser window.
Hype about Responsive Pages
In this device-oriented world we live in, web pages need to be both functional and aesthetically pleasing on all sized devies, from mobile phone screens to tablets to desktops. In fact, developers should be designing websites using 'mobile-first' design as mobile-phones overtook desktop computers as the most popular device worldwide for internet usage in 2014, and this figure continues to climb.
Semantic Structure
Semantic elements in HTML are those that have meaning, ie. tell us about the contents of that element. Some common semantc elements are:
- Headers -
h1, h2, h3... h6
- Images -
img
- Paragraphs -
p
- Forms -
form
- Tables -
table
- Articles -
article
Using Semantic Structure in your HTML means using these tags where they should be, to define the structure of those particular parts of your page.
My take on Frameworks and Responsive layouts is that using these allows a developer to see the big picture view of their page, and how all the elements on the page relate to each other. It is a straight-forward, no-mess solution to laying out a web page and ensuring the page will look great no matter what device it is viewed from!


Questions
Question1
Answer1
Question2
Answer2
Question3
Answer3
Question4
Answer4
Question5
Answer5
提交答案


A highly curated platform that brings together innovative founders with investors who offer value beyond just money.


All img
elements are given a max-width of 100% by default so that they will not overflow their containers. It is not necessary to apply any extra classes to make images responsive. You can observe responsive images in action by resizing the browser window as you look at the example below.


1 : /*
2 : * jcmaster.c
3 : *
4 : * Copyright (C) 1991-1997, Thomas G. Lane.
5 : * This file is part of the Independent JPEG Group's software.
6 : * For conditions of distribution and use, see the accompanying README file.
7 : *
8 : * This file contains master control logic for the JPEG compressor.
9 : * These routines are concerned with parameter validation, initial setup,
10 : * and inter-pass control (determining the number of passes and the work
11 : * to be done in each pass).
12 : */
13 :
14 : #define JPEG_INTERNALS
15 : #include "jinclude.h"
16 : #include "jpeglib.h"
17 :
18 :
19 : /* Private state */
20 :
21 : typedef enum {
22 : main_pass, /* input data, also do first output step */
23 : huff_opt_pass, /* Huffman code optimization pass */
24 : output_pass /* data output pass */
25 : } c_pass_type;
26 :
27 : typedef struct {
28 : struct jpeg_comp_master pub; /* public fields */
29 :
30 : c_pass_type pass_type; /* the type of the current pass */
31 :
32 : int pass_number; /* # of passes completed */
33 : int total_passes; /* total # of passes needed */
34 :
35 : int scan_number; /* current index in scan_info[] */
36 : } my_comp_master;
37 :
38 : typedef my_comp_master * my_master_ptr;
39 :
40 :
41 : /*
42 : * Support routines that do various essential calculations.
43 : */
44 :
45 : LOCAL(void)
46 3965: initial_setup (j_compress_ptr cinfo)
47 : /* Do computations that are needed before master selection phase */
48 : {
49 : int ci;
50 : jpeg_component_info *compptr;
51 : long samplesperrow;
52 : JDIMENSION jd_samplesperrow;
53 :
54 : /* Sanity check on image dimensions */
3945: branch 0 taken
20: branch 1 taken
3930: branch 2 taken
15: branch 3 taken
55 3965: if (cinfo->image_height <= 0 || cinfo->image_width <= 0
3930: branch 0 taken
0: branch 1 not taken
0: branch 2 not taken
3930: branch 3 taken
56 3930: || cinfo->num_components <= 0 || cinfo->input_components <= 0)
57 35: ERREXIT(cinfo, JERR_EMPTY_IMAGE);
58 :
59 : /* Make sure image isn't bigger than I can handle */
3893: branch 0 taken
37: branch 1 taken
43: branch 2 taken
3850: branch 3 taken
60 7823: if ((long) cinfo->image_height > (long) JPEG_MAX_DIMENSION ||
61 3893: (long) cinfo->image_width > (long) JPEG_MAX_DIMENSION)
62 80: ERREXIT1(cinfo, JERR_IMAGE_TOO_BIG, (unsigned int) JPEG_MAX_DIMENSION);
63 :
64 : /* Width of an input scanline must be representable as JDIMENSION. */
65 3850: samplesperrow = (long) cinfo->image_width * (long) cinfo->input_components;
66 3850: jd_samplesperrow = (JDIMENSION) samplesperrow;
0: branch 0 not taken
3850: branch 1 taken
67 3850: if ((long) jd_samplesperrow != samplesperrow)
68 0: ERREXIT(cinfo, JERR_WIDTH_OVERFLOW);
69 :
70 : /* For now, precision must match compiled-in value... */
0: branch 0 not taken
3850: branch 1 taken
71 3850: if (cinfo->data_precision != BITS_IN_JSAMPLE)
72 0: ERREXIT1(cinfo, JERR_BAD_PRECISION, cinfo->data_precision);
73 :
74 : /* Check that number of components won't exceed internal array sizes */
0: branch 0 not taken
3850: branch 1 taken
75 3850: if (cinfo->num_components > MAX_COMPONENTS)
76 0: ERREXIT2(cinfo, JERR_COMPONENT_COUNT, cinfo->num_components,
77 : MAX_COMPONENTS);
78 :
79 : /* Compute maximum sampling factors; check factor validity */
80 3850: cinfo->max_h_samp_factor = 1;
81 3850: cinfo->max_v_samp_factor = 1;
8148: branch 0 taken
3850: branch 1 taken
82 15848: for (ci = 0, compptr = cinfo->comp_info; ci < cinfo->num_components;
83 8148: ci++, compptr++) {
8148: branch 0 taken
0: branch 1 not taken
8148: branch 2 taken
0: branch 3 not taken
8148: branch 4 taken
0: branch 5 not taken
84 16296: if (compptr->h_samp_factor<=0 || compptr->h_samp_factor>MAX_SAMP_FACTOR ||
0: branch 0 not taken
8148: branch 1 taken
85 16296: compptr->v_samp_factor<=0 || compptr->v_samp_factor>MAX_SAMP_FACTOR)
86 0: ERREXIT(cinfo, JERR_BAD_SAMPLING);
87 8148: cinfo->max_h_samp_factor = MAX(cinfo->max_h_samp_factor,
88 : compptr->h_samp_factor);
89 8148: cinfo->max_v_samp_factor = MAX(cinfo->max_v_samp_factor,
90 : compptr->v_samp_factor);
91 : }
92 :
93 : /* Compute dimensions of components */
8148: branch 0 taken
3850: branch 1 taken
94 15848: for (ci = 0, compptr = cinfo->comp_info; ci < cinfo->num_components;
95 8148: ci++, compptr++) {
96 : /* Fill in the correct component_index value; don't rely on application */
97 8148: compptr->component_index = ci;
98 : /* For compression, we never do DCT scaling. */
99 8148: compptr->DCT_scaled_size = DCTSIZE;
100 : /* Size in DCT blocks */
101 8148: compptr->width_in_blocks = (JDIMENSION)
102 8148: jdiv_round_up((long) cinfo->image_width * (long) compptr->h_samp_factor,
103 8148: (long) (cinfo->max_h_samp_factor * DCTSIZE));
104 8148: compptr->height_in_blocks = (JDIMENSION)
105 8148: jdiv_round_up((long) cinfo->image_height * (long) compptr->v_samp_factor,
106 8148: (long) (cinfo->max_v_samp_factor * DCTSIZE));
107 : /* Size in samples */
108 8148: compptr->downsampled_width = (JDIMENSION)
109 8148: jdiv_round_up((long) cinfo->image_width * (long) compptr->h_samp_factor,
110 8148: (long) cinfo->max_h_samp_factor);
111 8148: compptr->downsampled_height = (JDIMENSION)
112 8148: jdiv_round_up((long) cinfo->image_height * (long) compptr->v_samp_factor,
113 8148: (long) cinfo->max_v_samp_factor);
114 : /* Mark component needed (this flag isn't actually used for compression) */
115 8148: compptr->component_needed = TRUE;
116 : }
117 :
118 : /* Compute number of fully interleaved MCU rows (number of times that
119 : * main controller will call coefficient controller).
120 : */
121 3850: cinfo->total_iMCU_rows = (JDIMENSION)
122 3850: jdiv_round_up((long) cinfo->image_height,
123 3850: (long) (cinfo->max_v_samp_factor*DCTSIZE));
124 3850: }
125 :
126 :
127 : #ifdef C_MULTISCAN_FILES_SUPPORTED
128 :
129 : LOCAL(void)
130 333: validate_script (j_compress_ptr cinfo)
131 : /* Verify that the scan script in cinfo->scan_info[] is valid; also
132 : * determine whether it uses progressive JPEG, and set cinfo->progressive_mode.
133 : */
134 : {
135 : const jpeg_scan_info * scanptr;
136 : int scanno, ncomps, ci, coefi, thisi;
137 : int Ss, Se, Ah, Al;
138 : boolean component_sent[MAX_COMPONENTS];
139 : #ifdef C_PROGRESSIVE_SUPPORTED
140 : int * last_bitpos_ptr;
141 : int last_bitpos[MAX_COMPONENTS][DCTSIZE2];
142 : /* -1 until that coefficient has been seen; then last Al for it */
143 : #endif
144 :
0: branch 0 not taken
333: branch 1 taken
145 333: if (cinfo->num_scans <= 0)
146 0: ERREXIT1(cinfo, JERR_BAD_SCAN_SCRIPT, 0);
147 :
148 : /* For sequential JPEG, all scans must have Ss=0, Se=DCTSIZE2-1;
149 : * for progressive JPEG, no scan can have this.
150 : */
151 333: scanptr = cinfo->scan_info;
333: branch 0 taken
0: branch 1 not taken
333: branch 2 taken
0: branch 3 not taken
152 666: if (scanptr->Ss != 0 || scanptr->Se != DCTSIZE2-1) {
153 : #ifdef C_PROGRESSIVE_SUPPORTED
154 333: cinfo->progressive_mode = TRUE;
155 333: last_bitpos_ptr = & last_bitpos[0][0];
585: branch 0 taken
333: branch 1 taken
156 918: for (ci = 0; ci < cinfo->num_components; ci++)
37440: branch 0 taken
585: branch 1 taken
157 38025: for (coefi = 0; coefi < DCTSIZE2; coefi++)
158 37440: *last_bitpos_ptr++ = -1;
159 : #else
160 : ERREXIT(cinfo, JERR_NOT_COMPILED);
161 : #endif
162 : } else {
163 0: cinfo->progressive_mode = FALSE;
0: branch 0 not taken
0: branch 1 not taken
164 0: for (ci = 0; ci < cinfo->num_components; ci++)
165 0: component_sent[ci] = FALSE;
166 : }
167 :
2502: branch 0 taken
333: branch 1 taken
168 2835: for (scanno = 1; scanno <= cinfo->num_scans; scanptr++, scanno++) {
169 : /* Validate component indexes */
170 2502: ncomps = scanptr->comps_in_scan;
2502: branch 0 taken
0: branch 1 not taken
0: branch 2 not taken
2502: branch 3 taken
171 2502: if (ncomps <= 0 || ncomps > MAX_COMPS_IN_SCAN)
172 0: ERREXIT2(cinfo, JERR_COMPONENT_COUNT, ncomps, MAX_COMPS_IN_SCAN);
3006: branch 0 taken
2502: branch 1 taken
173 5508: for (ci = 0; ci < ncomps; ci++) {
174 3006: thisi = scanptr->component_index[ci];
3006: branch 0 taken
0: branch 1 not taken
0: branch 2 not taken
3006: branch 3 taken
175 3006: if (thisi < 0 || thisi >= cinfo->num_components)
176 0: ERREXIT1(cinfo, JERR_BAD_SCAN_SCRIPT, scanno);
177 : /* Components must appear in SOF order within each scan */
504: branch 0 taken
2502: branch 1 taken
0: branch 2 not taken
504: branch 3 taken
178 3006: if (ci > 0 && thisi <= scanptr->component_index[ci-1])
179 0: ERREXIT1(cinfo, JERR_BAD_SCAN_SCRIPT, scanno);
180 : }
181 : /* Validate progression parameters */
182 2502: Ss = scanptr->Ss;
183 2502: Se = scanptr->Se;
184 2502: Ah = scanptr->Ah;
185 2502: Al = scanptr->Al;
2502: branch 0 taken
0: branch 1 not taken
186 2502: if (cinfo->progressive_mode) {
187 : #ifdef C_PROGRESSIVE_SUPPORTED
188 : /* The JPEG spec simply gives the ranges 0..13 for Ah and Al, but that
189 : * seems wrong: the upper bound ought to depend on data precision.
190 : * Perhaps they really meant 0..N+1 for N-bit precision.
191 : * Here we allow 0..10 for 8-bit data; Al larger than 10 results in
192 : * out-of-range reconstructed DC values during the first DC scan,
193 : * which might cause problems for some decoders.
194 : */
195 : #if BITS_IN_JSAMPLE == 8
196 : #define MAX_AH_AL 10
197 : #else
198 : #define MAX_AH_AL 13
199 : #endif
2502: branch 0 taken
0: branch 1 not taken
2502: branch 2 taken
0: branch 3 not taken
2502: branch 4 taken
0: branch 5 not taken
2502: branch 6 taken
0: branch 7 not taken
2502: branch 8 taken
0: branch 9 not taken
200 2502: if (Ss < 0 || Ss >= DCTSIZE2 || Se < Ss || Se >= DCTSIZE2 ||
2502: branch 0 taken
0: branch 1 not taken
2502: branch 2 taken
0: branch 3 not taken
0: branch 4 not taken
2502: branch 5 taken
201 2502: Ah < 0 || Ah > MAX_AH_AL || Al < 0 || Al > MAX_AH_AL)
202 0: ERREXIT1(cinfo, JERR_BAD_PROG_SCRIPT, scanno);
666: branch 0 taken
1836: branch 1 taken
203 2502: if (Ss == 0) {
0: branch 0 not taken
666: branch 1 taken
204 666: if (Se != 0) /* DC and AC together not OK */
205 0: ERREXIT1(cinfo, JERR_BAD_PROG_SCRIPT, scanno);
206 : } else {
0: branch 0 not taken
1836: branch 1 taken
207 1836: if (ncomps != 1) /* AC scans must be for only one component */
208 0: ERREXIT1(cinfo, JERR_BAD_PROG_SCRIPT, scanno);
209 : }
3006: branch 0 taken
2502: branch 1 taken
210 5508: for (ci = 0; ci < ncomps; ci++) {
211 3006: last_bitpos_ptr = & last_bitpos[scanptr->component_index[ci]][0];
1836: branch 0 taken
1170: branch 1 taken
0: branch 2 not taken
1836: branch 3 taken
212 3006: if (Ss != 0 && last_bitpos_ptr[0] < 0) /* AC without prior DC scan */
213 0: ERREXIT1(cinfo, JERR_BAD_PROG_SCRIPT, scanno);
95859: branch 0 taken
3006: branch 1 taken
214 98865: for (coefi = Ss; coefi <= Se; coefi++) {
37440: branch 0 taken
58419: branch 1 taken
215 95859: if (last_bitpos_ptr[coefi] < 0) {
216 : /* first scan of this coefficient */
0: branch 0 not taken
37440: branch 1 taken
217 37440: if (Ah != 0)
218 0: ERREXIT1(cinfo, JERR_BAD_PROG_SCRIPT, scanno);
219 : } else {
220 : /* not first scan */
58419: branch 0 taken
0: branch 1 not taken
0: branch 2 not taken
58419: branch 3 taken
221 58419: if (Ah != last_bitpos_ptr[coefi] || Al != Ah-1)
222 0: ERREXIT1(cinfo, JERR_BAD_PROG_SCRIPT, scanno);
223 : }
224 95859: last_bitpos_ptr[coefi] = Al;
225 : }
226 : }
227 : #endif
228 : } else {
229 : /* For sequential JPEG, all progression parameters must be these: */
0: branch 0 not taken
0: branch 1 not taken
0: branch 2 not taken
0: branch 3 not taken
0: branch 4 not taken
0: branch 5 not taken
0: branch 6 not taken
0: branch 7 not taken
230 0: if (Ss != 0 || Se != DCTSIZE2-1 || Ah != 0 || Al != 0)
231 0: ERREXIT1(cinfo, JERR_BAD_PROG_SCRIPT, scanno);
232 : /* Make sure components are not sent twice */
0: branch 0 not taken
0: branch 1 not taken
233 0: for (ci = 0; ci < ncomps; ci++) {
234 0: thisi = scanptr->component_index[ci];
0: branch 0 not taken
0: branch 1 not taken
235 0: if (component_sent[thisi])
236 0: ERREXIT1(cinfo, JERR_BAD_SCAN_SCRIPT, scanno);
237 0: component_sent[thisi] = TRUE;
238 : }
239 : }
240 : }
241 :
242 : /* Now verify that everything got sent. */
333: branch 0 taken
0: branch 1 not taken
243 333: if (cinfo->progressive_mode) {
244 : #ifdef C_PROGRESSIVE_SUPPORTED
245 : /* For progressive mode, we only check that at least some DC data
246 : * got sent for each component; the spec does not require that all bits
247 : * of all coefficients be transmitted. Would it be wiser to enforce
248 : * transmission of all coefficient bits??
249 : */
585: branch 0 taken
333: branch 1 taken
250 918: for (ci = 0; ci < cinfo->num_components; ci++) {
0: branch 0 not taken
585: branch 1 taken
251 585: if (last_bitpos[ci][0] < 0)
252 0: ERREXIT(cinfo, JERR_MISSING_DATA);
253 : }
254 : #endif
255 : } else {
0: branch 0 not taken
0: branch 1 not taken
256 0: for (ci = 0; ci < cinfo->num_components; ci++) {
0: branch 0 not taken
0: branch 1 not taken
257 0: if (! component_sent[ci])
258 0: ERREXIT(cinfo, JERR_MISSING_DATA);
259 : }
260 : }
261 333: }
262 :
263 : #endif /* C_MULTISCAN_FILES_SUPPORTED */
264 :
265 :
266 : LOCAL(void)
267 5341: select_scan_parameters (j_compress_ptr cinfo)
268 : /* Set up the scan parameters for the current scan */
269 : {
270 : int ci;
271 :
272 : #ifdef C_MULTISCAN_FILES_SUPPORTED
1824: branch 0 taken
3517: branch 1 taken
273 5341: if (cinfo->scan_info != NULL) {
274 : /* Prepare for current scan --- the script is already validated */
275 1824: my_master_ptr master = (my_master_ptr) cinfo->master;
276 1824: const jpeg_scan_info * scanptr = cinfo->scan_info + master->scan_number;
277 :
278 1824: cinfo->comps_in_scan = scanptr->comps_in_scan;
2214: branch 0 taken
1824: branch 1 taken
279 4038: for (ci = 0; ci < scanptr->comps_in_scan; ci++) {
280 2214: cinfo->cur_comp_info[ci] =
281 2214: &cinfo->comp_info[scanptr->component_index[ci]];
282 : }
283 1824: cinfo->Ss = scanptr->Ss;
284 1824: cinfo->Se = scanptr->Se;
285 1824: cinfo->Ah = scanptr->Ah;
286 1824: cinfo->Al = scanptr->Al;
287 : }
288 : else
289 : #endif
290 : {
291 : /* Prepare for single sequential-JPEG scan containing all components */
0: branch 0 not taken
3517: branch 1 taken
292 3517: if (cinfo->num_components > MAX_COMPS_IN_SCAN)
293 0: ERREXIT2(cinfo, JERR_COMPONENT_COUNT, cinfo->num_components,
294 : MAX_COMPS_IN_SCAN);
295 3517: cinfo->comps_in_scan = cinfo->num_components;
7563: branch 0 taken
3517: branch 1 taken
296 11080: for (ci = 0; ci < cinfo->num_components; ci++) {
297 7563: cinfo->cur_comp_info[ci] = &cinfo->comp_info[ci];
298 : }
299 3517: cinfo->Ss = 0;
300 3517: cinfo->Se = DCTSIZE2-1;
301 3517: cinfo->Ah = 0;
302 3517: cinfo->Al = 0;
303 : }
304 5341: }
305 :
306 :
307 : LOCAL(void)
308 5341: per_scan_setup (j_compress_ptr cinfo)
309 : /* Do computations that are needed before processing a JPEG scan */
310 : /* cinfo->comps_in_scan and cinfo->cur_comp_info[] are already set */
311 : {
312 : int ci, mcublks, tmp;
313 : jpeg_component_info *compptr;
314 :
3123: branch 0 taken
2218: branch 1 taken
315 5341: if (cinfo->comps_in_scan == 1) {
316 :
317 : /* Noninterleaved (single-component) scan */
318 3123: compptr = cinfo->cur_comp_info[0];
319 :
320 : /* Overall image size in MCUs */
321 3123: cinfo->MCUs_per_row = compptr->width_in_blocks;
322 3123: cinfo->MCU_rows_in_scan = compptr->height_in_blocks;
323 :
324 : /* For noninterleaved scan, always one block per MCU */
325 3123: compptr->MCU_width = 1;
326 3123: compptr->MCU_height = 1;
327 3123: compptr->MCU_blocks = 1;
328 3123: compptr->MCU_sample_width = DCTSIZE;
329 3123: compptr->last_col_width = 1;
330 : /* For noninterleaved scans, it is convenient to define last_row_height
331 : * as the number of block rows present in the last iMCU row.
332 : */
333 3123: tmp = (int) (compptr->height_in_blocks % compptr->v_samp_factor);
2955: branch 0 taken
168: branch 1 taken
334 3123: if (tmp == 0) tmp = compptr->v_samp_factor;
335 3123: compptr->last_row_height = tmp;
336 :
337 : /* Prepare array describing MCU composition */
338 3123: cinfo->blocks_in_MCU = 1;
339 3123: cinfo->MCU_membership[0] = 0;
340 :
341 : } else {
342 :
343 : /* Interleaved (multi-component) scan */
2218: branch 0 taken
0: branch 1 not taken
0: branch 2 not taken
2218: branch 3 taken
344 2218: if (cinfo->comps_in_scan <= 0 || cinfo->comps_in_scan > MAX_COMPS_IN_SCAN)
345 0: ERREXIT2(cinfo, JERR_COMPONENT_COUNT, cinfo->comps_in_scan,
346 : MAX_COMPS_IN_SCAN);
347 :
348 : /* Overall image size in MCUs */
349 2218: cinfo->MCUs_per_row = (JDIMENSION)
350 2218: jdiv_round_up((long) cinfo->image_width,
351 2218: (long) (cinfo->max_h_samp_factor*DCTSIZE));
352 2218: cinfo->MCU_rows_in_scan = (JDIMENSION)
353 2218: jdiv_round_up((long) cinfo->image_height,
354 2218: (long) (cinfo->max_v_samp_factor*DCTSIZE));
355 :
356 2218: cinfo->blocks_in_MCU = 0;
357 :
6654: branch 0 taken
2218: branch 1 taken
358 8872: for (ci = 0; ci < cinfo->comps_in_scan; ci++) {
359 6654: compptr = cinfo->cur_comp_info[ci];
360 : /* Sampling factors give # of blocks of component in each MCU */
361 6654: compptr->MCU_width = compptr->h_samp_factor;
362 6654: compptr->MCU_height = compptr->v_samp_factor;
363 6654: compptr->MCU_blocks = compptr->MCU_width * compptr->MCU_height;
364 6654: compptr->MCU_sample_width = compptr->MCU_width * DCTSIZE;
365 : /* Figure number of non-dummy blocks in last MCU column & row */
366 6654: tmp = (int) (compptr->width_in_blocks % compptr->MCU_width);
5034: branch 0 taken
1620: branch 1 taken
367 6654: if (tmp == 0) tmp = compptr->MCU_width;
368 6654: compptr->last_col_width = tmp;
369 6654: tmp = (int) (compptr->height_in_blocks % compptr->MCU_height);
5237: branch 0 taken
1417: branch 1 taken
370 6654: if (tmp == 0) tmp = compptr->MCU_height;
371 6654: compptr->last_row_height = tmp;
372 : /* Prepare array describing MCU composition */
373 6654: mcublks = compptr->MCU_blocks;
0: branch 0 not taken
6654: branch 1 taken
374 6654: if (cinfo->blocks_in_MCU + mcublks > C_MAX_BLOCKS_IN_MCU)
375 0: ERREXIT(cinfo, JERR_BAD_MCU_SIZE);
13308: branch 0 taken
6654: branch 1 taken
376 26616: while (mcublks-- > 0) {
377 13308: cinfo->MCU_membership[cinfo->blocks_in_MCU++] = ci;
378 : }
379 : }
380 :
381 : }
382 :
383 : /* Convert restart specified in rows to actual MCU count. */
384 : /* Note that count must fit in 16 bits, so we provide limiting. */
267: branch 0 taken
5074: branch 1 taken
385 5341: if (cinfo->restart_in_rows > 0) {
386 267: long nominal = (long) cinfo->restart_in_rows * (long) cinfo->MCUs_per_row;
266: branch 0 taken
1: branch 1 taken
387 267: cinfo->restart_interval = (unsigned int) MIN(nominal, 65535L);
388 : }
389 5341: }
390 :
391 :
392 : /*
393 : * Per-pass setup.
394 : * This is called at the beginning of each pass. We determine which modules
395 : * will be active during this pass and give them appropriate start_pass calls.
396 : * We also set is_last_pass to indicate whether any more passes will be
397 : * required.
398 : */
399 :
400 : METHODDEF(void)
401 7027: prepare_for_pass (j_compress_ptr cinfo)
402 : {
403 7027: my_master_ptr master = (my_master_ptr) cinfo->master;
404 :
3850: branch 0 taken
1491: branch 1 taken
1686: branch 2 taken
0: branch 3 not taken
405 7027: switch (master->pass_type) {
406 : case main_pass:
407 : /* Initial pass: will collect input data, and do either Huffman
408 : * optimization or data output for the first scan.
409 : */
410 3850: select_scan_parameters(cinfo);
411 3850: per_scan_setup(cinfo);
3850: branch 0 taken
0: branch 1 not taken
412 3850: if (! cinfo->raw_data_in) {
413 3850: (*cinfo->cconvert->start_pass) (cinfo);
414 3850: (*cinfo->downsample->start_pass) (cinfo);
415 3850: (*cinfo->prep->start_pass) (cinfo, JBUF_PASS_THRU);
416 : }
417 3850: (*cinfo->fdct->start_pass) (cinfo);
418 3850: (*cinfo->entropy->start_pass) (cinfo, cinfo->optimize_coding);
641: branch 0 taken
3209: branch 1 taken
419 7700: (*cinfo->coef->start_pass) (cinfo,
420 3850: (master->total_passes > 1 ?
421 : JBUF_SAVE_AND_PASS : JBUF_PASS_THRU));
422 3850: (*cinfo->main->start_pass) (cinfo, JBUF_PASS_THRU);
641: branch 0 taken
3209: branch 1 taken
423 3850: if (cinfo->optimize_coding) {
424 : /* No immediate data output; postpone writing frame/scan headers */
425 641: master->pub.call_pass_startup = FALSE;
426 : } else {
427 : /* Will write frame/scan headers at first jpeg_write_scanlines call */
428 3209: master->pub.call_pass_startup = TRUE;
429 : }
430 3850: break;
431 : #ifdef ENTROPY_OPT_SUPPORTED
432 : case huff_opt_pass:
433 : /* Do Huffman optimization for a scan after the first one. */
434 1491: select_scan_parameters(cinfo);
435 1491: per_scan_setup(cinfo);
243: branch 0 taken
1248: branch 1 taken
243: branch 2 taken
0: branch 3 not taken
0: branch 4 not taken
243: branch 5 taken
436 1491: if (cinfo->Ss != 0 || cinfo->Ah == 0 || cinfo->arith_code) {
437 1248: (*cinfo->entropy->start_pass) (cinfo, TRUE);
438 1248: (*cinfo->coef->start_pass) (cinfo, JBUF_CRANK_DEST);
439 1248: master->pub.call_pass_startup = FALSE;
440 1248: break;
441 : }
442 : /* Special case: Huffman DC refinement scans need no Huffman table
443 : * and therefore we can skip the optimization pass for them.
444 : */
445 243: master->pass_type = output_pass;
446 243: master->pass_number++;
447 : /*FALLTHROUGH*/
448 : #endif
449 : case output_pass:
450 : /* Do a data-output pass. */
451 : /* We need not repeat per-scan setup if prior optimization pass did it. */
0: branch 0 not taken
1929: branch 1 taken
452 1929: if (! cinfo->optimize_coding) {
453 0: select_scan_parameters(cinfo);
454 0: per_scan_setup(cinfo);
455 : }
456 1929: (*cinfo->entropy->start_pass) (cinfo, FALSE);
457 1929: (*cinfo->coef->start_pass) (cinfo, JBUF_CRANK_DEST);
458 : /* We emit frame/scan headers now */
438: branch 0 taken
1491: branch 1 taken
459 1929: if (master->scan_number == 0)
460 438: (*cinfo->marker->write_frame_header) (cinfo);
461 1929: (*cinfo->marker->write_scan_header) (cinfo);
462 1929: master->pub.call_pass_startup = FALSE;
463 1929: break;
464 : default:
465 0: ERREXIT(cinfo, JERR_NOT_COMPILED);
466 : }
467 :
468 7027: master->pub.is_last_pass = (master->pass_number == master->total_passes-1);
469 :
470 : /* Set up progress monitor's pass info if present */
0: branch 0 not taken
7027: branch 1 taken
471 7027: if (cinfo->progress != NULL) {
472 0: cinfo->progress->completed_passes = master->pass_number;
473 0: cinfo->progress->total_passes = master->total_passes;
474 : }
475 7027: }
476 :
477 :
478 : /*
479 : * Special start-of-pass hook.
480 : * This is called by jpeg_write_scanlines if call_pass_startup is TRUE.
481 : * In single-pass processing, we need this hook because we don't want to
482 : * write frame/scan headers during jpeg_start_compress; we want to let the
483 : * application write COM markers etc. between jpeg_start_compress and the
484 : * jpeg_write_scanlines loop.
485 : * In multi-pass processing, this routine is not used.
486 : */
487 :
488 : METHODDEF(void)
489 2473: pass_startup (j_compress_ptr cinfo)
490 : {
491 2473: cinfo->master->call_pass_startup = FALSE; /* reset flag so call only once */
492 :
493 2473: (*cinfo->marker->write_frame_header) (cinfo);
494 2473: (*cinfo->marker->write_scan_header) (cinfo);
495 2473: }
496 :
497 :
498 : /*
499 : * Finish up at end of pass.
500 : */
501 :
502 : METHODDEF(void)
503 5565: finish_pass_master (j_compress_ptr cinfo)
504 : {
505 5565: my_master_ptr master = (my_master_ptr) cinfo->master;
506 :
507 : /* The entropy coder always needs an end-of-pass call,
508 : * either to analyze statistics or to flush its output buffer.
509 : */
510 5565: (*cinfo->entropy->finish_pass) (cinfo);
511 :
512 : /* Update state for next pass */
2388: branch 0 taken
1248: branch 1 taken
1929: branch 2 taken
0: branch 3 not taken
513 5565: switch (master->pass_type) {
514 : case main_pass:
515 : /* next pass is either output of scan 0 (after optimization)
516 : * or output of scan 1 (if no optimization).
517 : */
518 2388: master->pass_type = output_pass;
1950: branch 0 taken
438: branch 1 taken
519 2388: if (! cinfo->optimize_coding)
520 1950: master->scan_number++;
521 2388: break;
522 : case huff_opt_pass:
523 : /* next pass is always output of current scan */
524 1248: master->pass_type = output_pass;
525 1248: break;
526 : case output_pass:
527 : /* next pass is either optimization or output of next scan */
1929: branch 0 taken
0: branch 1 not taken
528 1929: if (cinfo->optimize_coding)
529 1929: master->pass_type = huff_opt_pass;
530 1929: master->scan_number++;
531 1929: break;
532 : }
533 :
534 5565: master->pass_number++;
535 5565: }
536 :
537 :
538 : /*
539 : * Initialize master compression control.
540 : */
541 :
542 : GLOBAL(void)
543 3965: jinit_c_master_control (j_compress_ptr cinfo, boolean transcode_only)
544 : {
545 : my_master_ptr master;
546 :
547 3965: master = (my_master_ptr)
548 3965: (*cinfo->mem->alloc_small) ((j_common_ptr) cinfo, JPOOL_IMAGE,
549 : SIZEOF(my_comp_master));
550 3965: cinfo->master = (struct jpeg_comp_master *) master;
551 3965: master->pub.prepare_for_pass = prepare_for_pass;
552 3965: master->pub.pass_startup = pass_startup;
553 3965: master->pub.finish_pass = finish_pass_master;
554 3965: master->pub.is_last_pass = FALSE;
555 :
556 : /* Validate parameters, determine derived values */
557 3965: initial_setup(cinfo);
558 :
333: branch 0 taken
3517: branch 1 taken
559 3850: if (cinfo->scan_info != NULL) {
560 : #ifdef C_MULTISCAN_FILES_SUPPORTED
561 333: validate_script(cinfo);
562 : #else
563 : ERREXIT(cinfo, JERR_NOT_COMPILED);
564 : #endif
565 : } else {
566 3517: cinfo->progressive_mode = FALSE;
567 3517: cinfo->num_scans = 1;
568 : }
569 :
333: branch 0 taken
3517: branch 1 taken
570 3850: if (cinfo->progressive_mode) /* TEMPORARY HACK ??? */
571 333: cinfo->optimize_coding = TRUE; /* assume default tables no good for progressive mode */
572 :
573 : /* Initialize my private state */
0: branch 0 not taken
3850: branch 1 taken
574 3850: if (transcode_only) {
575 : /* no main pass in transcoding */
0: branch 0 not taken
0: branch 1 not taken
576 0: if (cinfo->optimize_coding)
577 0: master->pass_type = huff_opt_pass;
578 : else
579 0: master->pass_type = output_pass;
580 : } else {
581 : /* for normal compression, first pass is always this type: */
582 3850: master->pass_type = main_pass;
583 : }
584 3850: master->scan_number = 0;
585 3850: master->pass_number = 0;
641: branch 0 taken
3209: branch 1 taken
586 3850: if (cinfo->optimize_coding)
587 641: master->total_passes = cinfo->num_scans * 2;
588 : else
589 3209: master->total_passes = cinfo->num_scans;
590 3850: }
Generated: 2016-04-07 17:52 by zcov


Toggle navigation
Your Trips
Logout
Trips
Toggle navigation
Add Trip
Trip Name
Friend
Destination
Action
{% for trip in trips %}
{{ trip[1] }}
{{ trip[2] }}
{{ trip[3] }}
Toggle navigation
Delete trip
{% endfor %}


vignettes/flowr_overview.Rmd
flowr_overview.Rmd
setup()
This will copy the flowr helper script to ~/bin
. Please make sure that this folder is in your $PATH
variable. For more details refer to setup’s help section.
Running flowr from the terminal will fetch you the following:
Usage: flowr function [arguments]
status Detailed status of a flow(s).
rerun rerun a previously failed flow
kill Kill the flow, upon providing working directory
fetch_pipes Checking what modules and pipelines are available; flowr fetch_pipes
Please use 'flowr -h function' to obtain further information about the usage of a specific function.
Consider, a simple example where we have three instances of linux’s sleep
command. After its completion three tmp files are created with some random data. Then, a merging step follows, combining the tmp files into one big file. Next, we use du
to calculate the size of the merged file.
NGS context This is quite similar in structure to a typical workflow from where a series of alignment and sorting steps may take place on the raw fastq files. Followed by merging of the resulting bam files into one large file per-sample and further downstream processing.
To create this flow in flowr, we need the actual commands to run; and a set of instructions regarding how to stitch the individual steps into a coherent pipeline.
Here is a table with the commands we would like to run ( or flow mat
).
| samplename | jobname | cmd |
|---|---|---|
| sample1 | sleep | sleep 10 && sleep 2;echo ‘hello’ |
| sample1 | sleep | sleep 11 && sleep 8;echo ‘hello’ |
| sample1 | sleep | sleep 11 && sleep 17;echo ‘hello’ |
| sample1 | create_tmp | head -c 100000 /dev/urandom > sample1_tmp_1 |
| sample1 | create_tmp | head -c 100000 /dev/urandom > sample1_tmp_2 |
| sample1 | create_tmp | head -c 100000 /dev/urandom > sample1_tmp_3 |
| sample1 | merge | cat sample1_tmp_1 sample1_tmp_2 sample1_tmp_3 > sample1_merged |
| sample1 | size | du -sh sample1_merged; echo ‘MY shell:’ $SHELL |
Further, we use an additional file specifying the relationship between the steps, and also other resource requirements: flow_def.
| jobname | sub_type | prev_jobs | dep_type | queue | memory_reserved | walltime | cpu_reserved | platform | jobid |
|---|---|---|---|---|---|---|---|---|---|
| sleep | scatter | none | none | short | 2000 | 1:00 | 1 | local | 1 |
| create_tmp | scatter | sleep | serial | short | 2000 | 1:00 | 1 | local | 2 |
| merge | serial | create_tmp | gather | short | 2000 | 1:00 | 1 | local | 3 |
| size | serial | merge | serial | short | 2000 | 1:00 | 1 | local | 4 |
Note: Each row in a flow mat relates to one job. Jobname column is used to link flow definition with flow mat. Also, values in previous jobs (prev_jobs) are derived from jobnames.
We use the two files described above and stitch them to create a flow object
(which contains all the information we need for cluster submission).
ex = file.path(system.file(package = "flowr"), "pipelines")
flowmat = as.flowmat(file.path(ex, "sleep_pipe.tsv"))
flowdef = as.flowdef(file.path(ex, "sleep_pipe.def"))
fobj <- to_flow(x = flowmat,
def = flowdef,
flowname = "example1", ## give it a name
platform = "lsf") ## override platform mentioned in flow def
Refer to to_flow’s help section for more details.
We can use plot_flow to quickly visualize the flow; this really helps when developing complex workflows.
plot_flow(fobj) # ?plot_flow for more information
plot_flow(flowdef) # plot_flow works on flow definition as well
Refer to plot_flow’s help section for more details.
Dry run: Quickly perform a dry run, of the submission step. This creates all the folder and files, and skips submission to the cluster. This helps in debugging etc.
submit_flow(fobj)
Test Successful!
You may check this folder for consistency. Also you may re-run submit with execute=TRUE
~/flowr/sleep_pipe-20150520-15-18-27-5mSd32G0
Once, we have a flow we can submit it to the cluster using submit_flow
Submit to the cluster !
submit_flow(fobj, execute = TRUE)
Flow has been submitted. Track it from terminal using:
flowr status x=~/flowr/type1-20150520-15-18-46-sySOzZnE
Refer to submit_flow’s help section for more details.
Next, you may use status to monitor the status of a flow.
flowr status x=~/flowr/runs/sleep_pipe-20150520*
| | total| started| completed| exit_status| status|
|:---------|-----:|-------:|---------:|-----------:|---------:|
|001.sleep | 10| 10| 10| 0| completed|
|002.tmp | 10| 10| 10| 0| completed|
|003.merge | 1| 1| 1| 0| completed|
|004.size | 1| 1| 1| 0| completed|
Notice, how we skipped specifying the complete path. Status would try to use the basename and show status of any folder it can match. If there are multiple matched, status would show a summary of each.
Alternatively, to check a summarized status of several flows, use the parent folder. In this case the parent folder has 3 flows, and here is the summary:
flowr status x=~/flowr/runs
Showing status of: ~/flowr/runs
| | total| started| completed| exit_status| status|
|:---------|-----:|-------:|---------:|-----------:|---------:|
|001.sleep | 30| 30| 10| 0|processing|
|002.tmp | 30| 30| 10| 0|processing|
|003.merge | 3| 3| 1| 0| pending|
|004.size | 3| 3| 1| 0| pending|
Scalability: Quickly submit, and check a summarized OR detailed status on ten or hundreds of flows.
Refer to status’s help section for more details.
Incase something goes wrong, one may use to kill command to terminate all the relating jobs of a single flow OR multiple flows.
kill one flow:
flowr kill_flow x=flow_wd
One may instruct flowr to kill multiple flows, but flowr would confirm before killing.
flowr kill x='~/flowr/runs/sleep_pipe'
found multiple wds:
~/flowr/runs/sleep_pipe-20150825-16-24-04-0Lv1PbpI
~/flowr/runs/sleep_pipe-20150825-17-47-52-5vFIkrMD
Really kill all of them ? kill again with force=TRUE
To kill multiple flow, set force=TRUE:
kill x='~/flowr/runs/sleep_pipe*' force = TRUE
Refer to kill’s help section for more details.
flowr also enables you to re-run a pipeline in case of hardware or software failures.
rerun x=flow_wd start_from=<intermediate step>
rerun x=flow_wdmat = new_flowmat def = new_flowdef start_from=<intermediate step>
Refer to rerun’s help section for more details.
An easy and quick way to build a workflow is to create a set of two tab delimited files. First is a table with commands to run (for each step of the pipeline), while second has details regarding how the modules are stitched together. In the rest of this document we would refer to them as flow_mat
and flow_def
respectively (as introduced in the previous sections).
Let us read in examples of both these files to understand their structure.
ex = file.path(system.file(package = "flowr"), "pipelines")
flow_mat = as.flowmat(file.path(ex, "sleep_pipe.tsv"))
flow_def = as.flowdef(file.path(ex, "sleep_pipe.def"))
describes commands to run:
Each row in flow mat describes one shell command, with additional information regarding the name of the step etc.
Essentially, this is a tab delimited file with three columns:
samplename
: A grouping column. The table is split using this column and each subset is treated as an individual flow. Thus we may have one flowmat for a series of samples, and the whole set would be submitted as a batch.
jobname
: This corresponds to the name of the step. This should match exactly with the jobname column in flow_def table described below.cmd
: A shell command to run. One can get quite creative here. These could be multiple shell commands separated by a ;
or &&
, more on this here. Though to keep this clean you may just wrap a multi-line command into a script and just source the bash script from here.Here is an example flow_mat for the flowr described above.
| samplename | jobname | cmd |
|---|---|---|
| sample1 | sleep | sleep 10 && sleep 2;echo ‘hello’ |
| sample1 | sleep | sleep 11 && sleep 8;echo ‘hello’ |
| sample1 | sleep | sleep 11 && sleep 17;echo ‘hello’ |
| sample1 | create_tmp | head -c 100000 /dev/urandom > sample1_tmp_1 |
| sample1 | create_tmp | head -c 100000 /dev/urandom > sample1_tmp_2 |
| sample1 | create_tmp | head -c 100000 /dev/urandom > sample1_tmp_3 |
| sample1 | merge | cat sample1_tmp_1 sample1_tmp_2 sample1_tmp_3 > sample1_merged |
| sample1 | size | du -sh sample1_merged; echo ‘MY shell:’ $SHELL |
defines how to stitch pieces of the (work)flow:
Each row in this table refers to one step of the pipeline. It describes the resources used by the step and also its relationship with other steps, especially, the step immediately prior to it.
It is a tab separated file, with a minimum of 4 columns:
jobname
: Name of the stepsub_type
: Short for submission type, refers to, how should multiple commands of this step be submitted. Possible values are serial
or scatter
.prev_job
: Short for previous job, this would be jobname of the previous job. This can be NA/./none if this is a independent/initial step, and no previous step is required for this to start.dep_type
: Short for dependency type, refers to the relationship of this job with the one defined in prev_job
. This can take values none
, gather
, serial
or burst
.These would be explained in detail, below.
Apart from the above described variables, several others defining the resource requirements of each step are also available. These give great amount of flexibility to the user in choosing CPU, wall time, memory and queue for each step (and are passed along to the HPCC platform).
cpu_reserved
memory_reserved
nodes
walltime
queue
This is especially useful for genomics pipelines, since each step may use different amount of resources. For example, in other frameworks, if one step uses 16 cores these would be blocked and not used during processing of several other steps. Thus resulting in blockage of those cores. Flowr prevents this, by being able to tune resources granurly. Example, one may submit few short steps in short
queue, and longer steps of the same pipeline in say long
queue.
Most cluster platforms accept these resource arguments. Essentially a file like this is used as a template, and variables defined in curly braces ( ex. {{{CPU}}}
) are filled up using the flow definition file.
If these (resource requirements) columns are not included in the flow definition, their values should be explicitly defined in the submission template. One may customize the templates as described in the cluster support section.
Here is an example of a typical flow_def file.
| jobname | sub_type | prev_jobs | dep_type | queue | memory_reserved | walltime | cpu_reserved | platform | jobid |
|---|---|---|---|---|---|---|---|---|---|
| sleep | scatter | none | none | short | 2000 | 1:00 | 1 | local | 1 |
| create_tmp | scatter | sleep | serial | short | 2000 | 1:00 | 1 | local | 2 |
| merge | serial | create_tmp | gather | short | 2000 | 1:00 | 1 | local | 3 |
| size | serial | merge | serial | short | 2000 | 1:00 | 1 | local | 4 |
Let us use an example flow, to understand submission and dependency types.
Consider three steps A, B and C, where A has 10 commands from A1 to A10, similarly B has 10 commands B1 through B10 and C has a single command, C1. Consider another step D (with D1-D3), which comes after C.
step: A ----> B -----> C -----> D
# of cmds 10 10 1 3
This refers to the sub_type column in flow definition.
scatter
: submit all commands as parallel, independent jobs.
serial
: run these commands sequentially one after the other.
This refers to the dep_type column in flow definition.
none
: independent job.
serial
: one to one relationship with previous job.
gather
: many to one, wait for all commands in previous job to finish then start the current step.
burst
: one to many wait for the previous step which has one job and start processing all cmds in the current step.
Using the above submission and dependency types one can create several types of relationships between former and later jobs. Here are a few examples of relationships one may typically use.
A1 --------> B1
A2 --------> B1
.. --------> ..
A10 --------> B10
dependency submission dependency submission
none scatter serial scatter
relationship
ONE-to-ONE
Relationship between steps A and B is best defined as serial
. Step A (A1 through A10) is submitted as scatter. Further, \(i^th\) jobs of B depends on \(i^th\) jobs of A. i.e. B1 requires A1 to complete; B2 requires A2 and so on. Also, we note that defining dependency as serial, makes sure that B does not wait for all elements of A to complete.
B1 ----\
B2 -----\
.. -----> C1
B9 ------/
B10-----/
dependency submission dependency submission
serial scatter gather serial
relationship
MANY-to-ONE
Since C is a single command which requires all steps of B to complete, intuitively it needs to gather
pieces of data generated by B. In this case dep_type
would be gather
and sub_type
type would be serial
since it is a single command.
/-----> D1
C1 --------> D2
\-----> D3
dependency submission dependency submission
gather serial burst scatter
relationship
ONE-to-MANY
Further, D is a set of three commands (D1-D3), which need to wait for a single process (C1) to complete. They would be submitted as scatter
after waiting on C in a burst
type dependency.
In essence, an example flow_def would look like as follows (with additional resource requirements not shown for brevity):
ex2def = as.flowdef(file.path(ex, "abcd.def"))
ex2mat = as.flowmat(file.path(ex, "abcd.tsv"))
kable(ex2def[, 1:4])
| jobname | sub_type | prev_jobs | dep_type |
|---|---|---|---|
| A | scatter | none | none |
| B | scatter | A | serial |
| C | serial | B | gather |
| D | scatter | C | burst |
plot_flow(ex2def)
There is a darker more prominent shadow to indicate scatter steps.
As of now we have tested this on the following clusters:
| Platform | command | status | queue.type |
|---|---|---|---|
| LSF 7 | bsub | Beta | lsf |
| LSF 9.1 | bsub | Stable | lsf |
| Torque | qsub | Stable | torque |
| Moab | msub | Stable | moab |
| SGE | qsub | Beta | sge |
| SLURM | sbatch | alpha | slurm |
For more details, refer to the configuration section


We need both
closed
by None at 2021-07-21 13:09:16.possibly only needs cosmetic cleanup
there's a po/note_list in the test data, but it's not populated
going with notificationdata/vendornote as the most plausible candidate
not seeing fields for adjustments or prepay, but Paul advises they're not needed
assigned
by None at 2021-07-21 13:09:57.I will check my local dump for these fixtures and if I find them I will check the marc record to see if there is bada data in 852 or if traject is not doing something right.
labeled
by Christina Chortaria at 2021-07-21 12:14:36.labeled
by Christina Chortaria at 2021-07-21 12:14:36.assigned
by Christina Chortaria at 2021-07-21 13:30:31.@mzelesky @hackartisan can you review again?
review_requested
by Christina Chortaria at 2021-07-21 14:32:30.review_requested
by Christina Chortaria at 2021-07-21 14:39:08.mentioned
by None at 2021-07-21 14:40:28.subscribed
by None at 2021-07-21 14:40:29.mentioned
by None at 2021-07-21 14:40:29.subscribed
by None at 2021-07-21 14:40:29.deployed
by Trey Pendragon at 2021-07-21 15:34:18.closed
by Eliot Jordan at 2021-07-21 14:53:01.referenced
by Eliot Jordan at 2021-07-21 14:53:01.merged
by Eliot Jordan at 2021-07-21 14:53:01.closed
by Eliot Jordan at 2021-07-21 14:53:01.head_ref_deleted
by Eliot Jordan at 2021-07-21 14:53:04.labeled
by None at 2021-07-21 23:27:10.moved_columns_in_project
by Eliot Jordan at 2021-07-21 16:50:39.assigned
by Eliot Jordan at 2021-07-21 16:51:05.Dupe. Closed by https://github.com/pulibrary/figgy/pull/4582
closed
by Eliot Jordan at 2021-07-21 20:05:15.closed
by None at 2021-07-21 20:40:10.moved_columns_in_project
by None at 2021-07-21 20:40:11.referenced
by None at 2021-07-21 20:40:10.merged
by None at 2021-07-21 20:40:10.closed
by None at 2021-07-21 20:40:10.head_ref_deleted
by None at 2021-07-21 20:40:11.head_ref_force_pushed
by None at 2021-07-21 14:43:44.The jobs need to be turned on on different days. We can not jsut turn them all on at once
closed
by None at 2021-07-21 14:59:48.@mzelesky What do we want to do about invalid invoices? Should I only mark off the valid ones, since we are expecting the invalid ones to be sent again at a later date?
This is not for the voucher feed, but the fund transactions.
There is no need to track invoice ids and look for duplicate invoices. This ticket is for the files posted to lbvchr-p/fund_adjustment
assigned
by None at 2021-07-21 15:31:14.mentioned
by None at 2021-07-21 17:38:50.subscribed
by None at 2021-07-21 17:38:50.referenced
by None at 2021-07-21 18:13:11.deployed
by None at 2021-07-21 18:14:46.deployed
by None at 2021-07-21 20:00:44.referenced
by None at 2021-07-21 20:13:19.merged
by None at 2021-07-21 20:13:19.closed
by None at 2021-07-21 20:13:19.head_ref_deleted
by None at 2021-07-21 20:13:20.referenced
by Zaynab Masood at 2021-07-21 20:25:14.merged
by Zaynab Masood at 2021-07-21 20:25:14.closed
by Zaynab Masood at 2021-07-21 20:25:15.head_ref_deleted
by Zaynab Masood at 2021-07-21 20:25:17.referenced
by Bess Sadler at 2021-07-21 13:13:03.merged
by Bess Sadler at 2021-07-21 13:13:03.closed
by Bess Sadler at 2021-07-21 13:13:03.head_ref_deleted
by Bess Sadler at 2021-07-21 13:13:06.https://github.com/pulibrary/oawaiver/pull/12 should have this resolved, but I also wanted to ensure that Dependabot alerts were enabled on https://github.com/pulibrary/oawaiver/settings/security_analysis before closing this.
closed
by James R. Griffin III at 2021-07-21 13:23:58.As of 09:24, members of OIT have assisted me with resetting the RSA soft token needed to access the server with the legacy application:
❯ ssh<EMAIL_ADDRESS>[...]
Enter PASSCODE:
[...]
IMPORTANT EMAIL ADDRESS
All Questions<EMAIL_ADDRESS>libvijrg@eisess204l:~ $
I have attempted to ensure that I am using the correct legacy credentials for the system user oar
, however, it now seems to be the case that either these credentials have been updated or, more likely, that password-only authentication over the SSH has been disabled. I have submitted a support request to OIT in order to advance this:
assigned
by Bess Sadler at 2021-07-21 13:14:20.unassigned
by Bess Sadler at 2021-07-21 15:34:36.review_requested
by James R. Griffin III at 2021-07-21 17:01:18.referenced
by Bess Sadler at 2021-07-21 17:20:03.merged
by Bess Sadler at 2021-07-21 17:20:03.closed
by Bess Sadler at 2021-07-21 17:20:03.head_ref_deleted
by Bess Sadler at 2021-07-21 17:20:04.I am going to consider this scoped for running the RSpec test suites, as well as RuboCop.
assigned
by James R. Griffin III at 2021-07-21 14:00:21.assigned
by James R. Griffin III at 2021-07-21 17:23:19.labeled
by None at 2021-07-21 17:20:40.referenced
by Bess Sadler at 2021-07-21 17:25:31.merged
by Bess Sadler at 2021-07-21 17:25:31.closed
by Bess Sadler at 2021-07-21 17:25:31.head_ref_deleted
by Bess Sadler at 2021-07-21 17:25:31.closed
by Francis Kayiwa at 2021-07-21 18:05:51.assigned
by James R. Griffin III at 2021-07-21 20:02:28.assigned
by James R. Griffin III at 2021-07-21 23:15:33.assigned
by Christina Chortaria at 2021-07-21 15:39:28.closed
by None at 2021-07-21 15:56:53.referenced
by Christina Chortaria at 2021-07-21 16:35:52.merged
by Christina Chortaria at 2021-07-21 16:35:52.closed
by Christina Chortaria at 2021-07-21 16:35:52.head_ref_deleted
by Christina Chortaria at 2021-07-21 16:35:53.Coverage increased (+0.07%) to 96.813% when pulling f676375313370103905fd32dfcffcb79d783babf on alma into 9564b07d6d648faedeb7501dac59d8fac711093d on main.
deployed
by Trey Pendragon at 2021-07-21 15:16:53.deployed
by Trey Pendragon at 2021-07-21 15:56:58.deployed
by Trey Pendragon at 2021-07-21 16:52:25.review_requested
by Francis Kayiwa at 2021-07-21 14:10:29.renamed
by Trey Pendragon at 2021-07-21 17:26:20.referenced
by Eliot Jordan at 2021-07-21 17:50:56.merged
by Eliot Jordan at 2021-07-21 17:50:56.closed
by Eliot Jordan at 2021-07-21 17:50:56.head_ref_deleted
by Eliot Jordan at 2021-07-21 17:50:56.This will close this https://github.com/pulibrary/oawaiver/issues/7
review_requested
by Francis Kayiwa at 2021-07-21 14:58:39.head_ref_force_pushed
by Francis Kayiwa at 2021-07-21 15:06:23.ready_for_review
by Francis Kayiwa at 2021-07-21 16:47:15.referenced
by Bess Sadler at 2021-07-21 19:03:19.merged
by Bess Sadler at 2021-07-21 19:03:19.closed
by Bess Sadler at 2021-07-21 19:03:20.head_ref_deleted
by Bess Sadler at 2021-07-21 19:03:20.referenced
by Bess Sadler at 2021-07-21 19:04:18.merged
by Bess Sadler at 2021-07-21 19:04:18.closed
by Bess Sadler at 2021-07-21 19:04:18.head_ref_deleted
by Bess Sadler at 2021-07-21 19:04:18.referenced
by Francis Kayiwa at 2021-07-21 20:03:56.merged
by Francis Kayiwa at 2021-07-21 20:03:56.closed
by Francis Kayiwa at 2021-07-21 20:03:56.head_ref_deleted
by Francis Kayiwa at 2021-07-21 20:03:57.head_ref_force_pushed
by None at 2021-07-21 20:25:42.referenced
by Francis Kayiwa at 2021-07-21 23:08:31.merged
by Francis Kayiwa at 2021-07-21 23:08:31.closed
by Francis Kayiwa at 2021-07-21 23:08:31.head_ref_deleted
by Francis Kayiwa at 2021-07-21 23:08:32.head_ref_force_pushed
by Francis Kayiwa at 2021-07-21 23:09:15.labeled
by None at 2021-07-21 12:03:32.labeled
by None at 2021-07-21 12:03:32.labeled
by Kevin Reiss at 2021-07-21 17:28:17.renamed
by Kevin Reiss at 2021-07-21 17:28:31.review_requested
by Kevin Reiss at 2021-07-21 23:55:27.mentioned
by None at 2021-07-21 23:55:27.subscribed
by None at 2021-07-21 23:55:27.Resolved: Dataspace is back up in 325f4de
.
labeled
by Francis Kayiwa at 2021-07-21 18:09:47.labeled
by Francis Kayiwa at 2021-07-21 18:09:47.locked
by Francis Kayiwa at 2021-07-21 18:09:48.closed
by Francis Kayiwa at 2021-07-21 18:34:38.