{% load static %}
Keeping It Real, Saying It Like It Is
Born and raised in a small town in Kwazulu Natal called Pietermaritzburg, a poor but humble background with all odds against me and my family, I had to face challenges daily and deal with them. At the tender young age of about
10, I had the opportunity of going out every morning, very early with my siblings to sell Samoosa's and Murukku that my mum had made.
With the money we made, we could buy bread, which our lunch was made from. Some days we had no sales, which meant no lunch. During the latter part of my school years, I worked part time as a PA (PA = Petrol Attendant/Bowser
Boy) and at a grocery store to earn extra cash to help my family. In 1989, I completed my Grade 12 but did not pass.   Thank God that within my community, people did not bother much about my situation as they had their own problems.
...
I slowly realised that I needed to be thankful and grateful,
as it was a lesson well learnt. Besides, my parents did not
have the funds for me to further my education, life was tough and this would
have drained them more. God has always been by my side, the same year I got
employment at Lipton Tea as a Packer/Sweeper. I took on
the job with open arms just to get a salary and help my parents. I DID WHAT
IT TOOK! It was here that I began to realise that this is not what I wanted
to do for the rest of my Life. The tough conditions made
me stronger and more motivated to do BETTER with my Life. I started working
extra time to get more money and study while working. This lead to me moving
between various positions within the company until I transferred
to Unilever Boksburg in the year 2000 as a Technical Engineering Buyer.
This was a Game Changer for me, I worked extremely hard and self-studied which
lead to me getting one of the highest qualifications in
the IT world...I became SAP Certified, all on my own hard work and
perseverance. The reason why I say this was a Game Changer is that it
allowed me to leave my comfort zone as a permanent employee into a
contractual role, yes working for myself.
This also got tough at times but the freedom
outweighed anything else. Prior to me being SAP Certified, I also got my
Production Management and Purchasing/Supply Management Diploma’s.
I was still not satisfied and had loads of energy in ME! I decided to
venture further into the MLM - Multi-Level Marketing space and was
successful in the three programs I did. They say Life happens and YES,
it brought me down to size once again in 2015 when I had no work for over a
year and a half. This was one of my biggest life challenges ever as I found
myself unable to provide for my family.
This was tough and I pressurised myself even further because I felt hopeless. I
thank God
for my wife Seema who has been a pillar of strength and support to me, and
my children who stood by me during this time. The biggest
lesson I learnt from this was that when you fall down...lying on your
back...you are still looking up to the sky and one should not give up so
easily. I decided I AM NEVER GOING TO GIVE UP! I kept my prayer
and my sanity intact, and started to pick myself up again. In 2015, I also
founded #straighttalkwithnolan but did not do much with it although I had
the time.
I had the ideas and the plan with the project but
had no heart for some reason to pursue it. Life did get better, I was called
by a friend and offered at position at one of the biggest companies globally
ABInBEV. I accepted the opportunity because I needed to put food on the table and
make sure my family has a place they can call
home. Forward wind to 2018, #straighttalkwithnolan is back in action and
will be officially launching in January 2019.
This is the most exciting project I have ever worked on because it allows me to work
on
MYSELF first and then help others work on THEMSELVES! This program is based
on the daily challenges we face and how we can handle them
better. We are all individuals in our own right and we owe it to ourselves
to be the best version of ourselves. No one is going to change our
mind-sets, we have to adapt and make the decision alone. This is
where my program will help you make those hard felt decisions.
I have been involved in many self-development programs like Millionaire Mind
Intensive,
Enlightened Warrior Power, Train the Trainer and often attend
motivational talks. Now is the time to give back to Africa. My Mission and
Vision: To Change Lives in Africa and then Globally through Education,
Knowledge and Empowerment. My Personal Motto – Go Forth and Conquer!
My Mt Kilimanjaro journey - Scaling the heights of success! Summitting Africa's highest point and the World's highest free-standing mountain was no easy challenge, but the quest to raise awareness and funding for girls who are in need of sanitary pads was well worth the training and the risk of our lives. The journey was done as part of the Trek-4-Mandela expedition, in partnership with Imbumba Foundation. Our journey started on the 13th July and ended on Tata Madiba's birthday. Summit night meant leaving camp at 11pm on the 17th July and summitting at 10:30am on Madiba's birthday. That meant 11 and a half hours of walking in the cold, windy night until sunrise he next morning. The feeling overall was priceless. Mission accomplished.
The Enlightened Warrior Program was a three day 10 hour plus program. It covered "How to Access your True Power and Succeed in Spite of Anything". For many, life has become more about what they can’t do rather than what they can do. The truth, however, is that you are far greater than you think and have truly amazing potential. The key is to tap into your higher self and unleash your full capacity for success and happiness. The picture with me represents a steel rod which my partner and I bent just by understanding how powerful our mind is.
The Millionaire Mind Intensive program was a 3-day in-person course. It is designed to teach you proven ways to manage your money effectively, create financial opportunities for more income, and overcome the limiting thoughts, beliefs and feelings that keep you from the wealth, abundance and freedom you deserve and desire. I attended the event with my wife Seema and we were both astounded at him we limit our minds daily. The event for me personally was a game changer and I have since discovered areas in my life that have been holding me back financially and learnt ways to change them - right there, right then!
Rise Against Hunger meal packaging event is a fun and engaging way for volunteers to be part of the drive to end child hunger. Rise Against Hunger is a programme of meal packaging events, where volunteers package the meals. The meals are highly nutritious and comprise of rice, soya, dehydrated vegetable mix and a fortification pack of 23 essential minerals and vitamins specifically formulated to combat malnutrition (complying with the UNICEF standard).
The candidates chosen to pilot the Be the BEST Version of YOURSELF program were hand-picked by Nolan Pillay for their highly inspirational stories. He knew that their incredible stories of success against such difficult odds had to be told because of how they could inspire others never to give up on their dreams. Meet these phenomenal candidates selected by Nolan and learn more about their amazing journeys here...
I'm Pranisha, 34 years and mother to 2 wonderful girls. I am currently a teacher, teaching Chinese people how to speak English. I had also taught at schools both primary and secondary. Teaching was never ever a thought of mine, in fact I wanted to go into corporate or be a police officer. Before this stage of my life, I went through a very turbulent phase in my life which started in my early 20s going up to the age of 30. This was due to life choices that I made and also, I felt I was just attracting bad things to happen to me. I have been through heavy depressions, suicidal behavior (before my children were born), and neglecting myself which caused my life to become stagnant. At this time, I was working for a cellular company which basically wore me down and was not bringing me any joy. This is when I met my then husband, married and had my two girls. I felt no joy in my life even after marriage due to marital problems which resulted in a failed marriage that just added to my depression. I had left work when my little one was born, and life was not easy as finances were running out. I was filled with so much negativity and felt that there was this huge void in me, and I cried every day, especially now with a marriage that turn toxic and was becoming abusive. Living in misery for so long, I looked at my children and realized I need to make a change for them to show them what they need to be happy. The first thing I did was separate from a toxic relationship, so I left. The next step required me to change my whole mindset, to train my mind to see the good and be positive. This required me to do things that will empower me and help me mentally. I started exercising, partaking in races, stopping myself from any negative thoughts and building myself up physically and mentally. It was in this time that I realized that I wanted to also do something to also motivate and advise others in helping them out of their depressions. So, I decided to become a teacher as I felt that this was the best platform to make a difference. I then started studying again and got a job as an online teacher which helped so much. 4 years down the line this year I graduated with my degree and being a single mum to my kids. The moral of my story is that anything can be achieved when you are determined, and when you are determined there is no option for failure. The energy you put out is what you get back so if you are positive and see the good in everything and learn from the bad, you will be able to pick yourself up every time and make yourself a better person. Any situation even if it's one that you feel where you are sinking so deep and can’t get out, you can get out and turn it around. I have just given you a mild version of my story because there have been disturbing times but all it does is make you stronger. I am not where I want to be, but I'm very far from where I used to be, I have never felt more happier with myself. I have built myself up and am teaching my children how to do so too and now that I'm in my best place to date, I want to help others and take them out of their dark and difficult times and help them realize that they have all the power inside of them. Thank you
Jabulane Hope Thabethe (24) is a philanthropist from Johannesburg Katlehong Gauteng who established Barefoot Walk (Walk for change) under the Hope for Africa foundation which also discovered in 2016? Hope for Africa Foundation was driven by his Mother's Death which left him dry& Hopeless, after feeling this way, he thought of how many kids are going through hopelessness and loneliness without anyone to guide nor even show them love. Barefoot Walk initiative was and is driven by the increasing number of kids who walk long distances to school others barefooted others with improper school shoes not because they like but because of the situation , tolerating and getting used to being laughed at because they have a dream& a mission to accomplish which is to keep going with this initiative , he want to get into depth with it& Provide Full school uniform so to give a little motivation to kids. 2016 & 2017 He walked from Katlehong east of Johannesburg to Durban raising school shoes and sanitary pads. He has already taken this out from South Africa to our neighbouring Countries Lesotho, Botswana & East Africa Kenya, so far as the foundation have helped over 3862 kids across South Africa and Africa Restoring humanity. So, the next generation will not watch one another giving up on their dreams because of not having simple things. REBUILDING THE SPIRIT OF SHARING Recently he was among the top 100 Shinning stars in City of Ekurhuleni and he is currently an advocate of peace appointed as a assistant country director at global peace chain South Africa and 2016 won SA upcoming Leader of the year at SA Awards.
 Born in Namibian and will always be one, it’s in my blood, love the beautiful open spaces, the cleanliness of my country and all its friendly
people. I’ve lived in South Africa for the past 24 years. Founder and Director of Le’JFab inspired by fabulous women, owner of the company PFT Creations establish 2016, and a member of Glow Getters establish January
2018. Women empowerment movement. I offer my time and assist to community projects and are part of my local Spar community project to assist them collecting blankets, food parcels and go out to Thembisa for a soup
kitchen and volunteer my assistance to these projects. I’m currently have a campaign running, Heels & Jeans with a twist of Kizomba in the mix and have a lot of love for this dance. I truly believe it brings out
the “Inner Goddess” we all have as women. I’m a true believer in myself and want to make people feel wonderful about themselves, love yourself and be kind to yourself. Live your life to the fullest, make a statement
in life that people can remember you not for what you give them but the impact you made in their lives. The lessons you learn and the journey you went through. That’s my Motto in life. I draw inspiration from other
women and stand to empower women. I believe women have so much power and if we stand together, how much power we share and could do if we unify as ONE. My background is that I hold a Diploma in Marketing, Human
Resources and work for 6 years as an events and marketing co coordinator at Coca Cola Namibia Beverages’ before I relocated to SA in 1997. I still do have family and lots of friends that keep me grounded. I’ve worked
for big corporates’ likes of Alexander Forbes, Anon, Ernst & Young, Sasol, ABSA, Health Care Consultants and Eskom. I’m currently employed as the Office & Human Resources Manager for an international company Makon
Group, but my passion is still events and Marketing. I love it with all my Heart.
Quote: “You will find true success and Happiness, if you have only ONE goal” To fulfil the highest most truthful expression of yourself as a human being.  - Oprah Winfrey
Life has challenged me so much, I am a young man who has faced a lot in life. I have a lot to tell but I will be brief. When I was a young boy, I lost four family members my father, mother, brother and older sister within the period of 5 years. I did not get a foster care or social grant due to family conflicts.Me and my young sister, we did not have any income in our home, but we had to take care of my older sister's children that she left when she died. My young sister had to look for a job so that we could eat.It was very hard for me to continue with the school because I had to beg for the school uniform from other children, knowing that you will go to school hungry and come back still not having something to eat is very hard. Some days we were going to bed without eating anything.I decided that I will finish the school and go to look for jobs because I am the only hope my family has. After getting better result in my matric I went to WITS university to do computer science that where I met the guy called Nolan. when I met Nolan, I was: Afraid of facing the real world I thought I was not strong enough, I was confused about my life I did not know if I should go for jobs so that I can put a bread in the table for my family or must I continue with school. I was afraid to tell my family that I do not like computer science because they thought It the best career for me. I had the low self- esteem and low self-confidence I thought I could not speak English as I learnt in rural Areas.  Even now I am still moving because he keeps on chasing me, he keeps on motivating me that why I am still moving. I am a better person and I decided to go on with my education, I am living a positive life now and I can help others. I am using Nolan’s daily quotes to face daily challenges, they are very powerful.
Muna Mogorosi is an entrepreneur, founder and CEO of Net-Rising Projects and FinTech called MoyaPay, a leadership coach and inspiration speaker. He is known for his positive influence that he possesses on people he encounters with, the ability to speak, touch and inspire individuals’ lives. He has learned from local and international high-ranking professionals in their respective fields such as Dr John Demartini, human behaviour specialist, JT Foxx, worlds #1 wealth coach, Robert Kiyosaki, author of Rich Dad Poor Dad and Robin Banks, the Mind power coach just to name a few. His one of the 4th Industrial Revolution (4IR) pioneers, Muna`s favourite saying is: In order to live a meaningful and purposeful life, you need to give yourself permission to do something extraordinary with your life, add value in peoples` lives and the universe will reward you.
The Be the BEST Version of YOURSELF program has been compiled by a panel of professionals who have years of specialist experience under their belts. Each contributing team member has strategically formulated their respective topics to ensure that you receive the most effective benefits of each Value as presented in the program. Meet our highly respected panel of experts here...
Nolan Pillay is the brilliant and dynamic founder of South Africa’s first ever trademarked self-development programme: Be the BEST Version of YOURSELF ™. Rivalling the likes of similar international programs that have flooded the country, Nolan’s life-altering system has more than just one homegrown advantage to offer. A born South African, Nolan has an almost instinctive knowledge of the diverse pulses that make up the heartbeat of the African continent; and most importantly, he’s been there. He’s the man that has seen the harsh side of poverty, who knows the pangs of hunger as a helpless child, who has never given up on his dreams and who is ready to share his success story with the world – if only for the purpose of empowering others to achieve the same.
PD Devnarain is a practising Reiki Master and Healer for the last 12 years. Her spiritual journey began at a young age when she became aware of the existence of a spiritual realm outside of the 3rd dimension through various incidents which could only be explained as supernatural ones. Following the path of light and healing, PD`s healing work began with her study of Reiki – a Japanese technique which brings about changes ranging from merely a deep sense of relaxation all the way to profound existential changes in the lives of one`s clients. Additional research, studies and personal experiences in spirituality over the years have taught her how to bring about a new form of conscious healing within clients in conjunction with the various modalities that she works with. PD also holds a Communications Degree, majoring in Communications, English and Psychology. She is also an entrepreneur who owns a PR, Marketing and Communications company called PR Divas, based in Johannesburg, South Africa.
Adam Steyl is a maverick, magnetic and supercharged serial entrepreneur. Within a short time frame, he has been able to meet and learn from some of the biggest authorities in the business world such as JT Foxx, the world’s #1 wealth coach, Robert Kiyosaki, author of the Rich dad series, and billionaire property tycoon, Hugh Hilton. Adams mission statement is “Creating magnetic success”, and he has the results to prove it. He`s a Focus and Wealth strategist, an authentic speaker and coach who is driven to not only share his wealth of knowledge but also some of the lessons learned during the road of personal growth and self-development that got him to where he is today.
Muna Mogorosi is an entrepreneur, founder and CEO of Net-Rising Projects and FinTech called MoyaPay, a leadership coach and inspiration speaker. He is known for his positive influence that he possesses on people he encounters with, the ability to speak, touch and inspire individuals’ lives. He has learned from local and international high-ranking professionals in their respective fields such as Dr John Demartini, human behaviour specialist, JT Foxx, worlds #1 wealth coach, Robert Kiyosaki, author of Rich Dad Poor Dad and Robin Banks, the Mind power coach just to name a few. His one of the 4th Industrial Revolution (4IR) pioneers, Muna`s favourite saying is: In order to live a meaningful and purposeful life, you need to give yourself permission to do something extraordinary with your life, add value in peoples` lives and the universe will reward you.
“Thank you, Nolan for your commitment and guidance to Molefe and your mentorship program, that has given this young man much belief and confidence in his capabilities of reaching, his goals and aspirations.”
“Molefe joined the class on the 26th of February 2018. From then onwards, he was never absent from school, unless on the days of him collecting his medication at the Clinic. (for Epilepsy) He was enthusiastic and eager to learn more in his Mathematics and Physical Science Lessons. No wonder that in our last revision before the final exam, he topped the class. I strongly believe that this mentorship course helped motivate him, in life in general, as well as in his commitment to his studies. ”
“His motivation and goal is to open a business, where he can employ others. I have every confidence that with the encouragement, skills and leadership and mentorship guidance, he has received from Nolan and the mentorship program, he will succeed. Here is a true leader and his success is in the making. ”
FOLLOW US


Before Change
def encode_sample(self, image_or_path):
Convert the given image into a dict convertible to tf example.
if isinstance(image_or_path, six.string_types):
// TODO(epot): np_image = load_image_from_disk(image_or_path)
raise NotImplementedError
elif isinstance(image_or_path, np.ndarray):
np_image = image_or_path
else:
// Could also add PIL support
raise ValueError("Could not convert {} to image".format(image_or_path))
// Check that the image is valid
if np_image.dtype != np.uint8:
raise ValueError("Image should be uint8. Detected: {}".format(
np_image.dtype))
utils.assert_shape_match(np_image.shape, self._shape)
After Change
encoded_image = self._encode_image(image_or_path)
else:
with tf.gfile.Open(image_or_path, "rb") as image_f:
encoded_image = image_f.read()
return encoded_image
def decode_sample(self, sample):
Reconstruct the image from the tf example.


Free body diagrams are a technique that helps visualize the forces on a mass in order to solve Newton's second law.
A free body diagram shows all the force vectors on a single mass. Mass is typically written inside a box, with force vectors pointed away from the box.
solution
$$\sum F=ma$$ $$-200\,\mathrm{N}+300\,\mathrm{N}+500\,\mathrm{N}=ma$$ $$600\,\mathrm{N}=(2\,\mathrm{kg})a$$ $$300 \mathrm{\tfrac{m}{s^2}}=a$$solution
$$\sum F=ma$$ $$F_{N} + F_{g} = ma$$ $$F_{N} - 0.34\, \mathrm{N} = (0.09\, \mathrm{kg})(0)$$ $$F_{N} - 0.34\, \mathrm{N} = 0$$ $$F_{N} = 0.34 \, \mathrm{N}$$solution
solution
$$\sum F=ma$$ $$F_1+F_2=ma$$ $$-833+663 = m(2)$$ $$\frac{-170}{2} = \frac{m(2)}{2}$$ $$-85 \, \mathrm{kg} = m$$A negative mass doesn't make sense. Where did we mess up?
Oh, the acceleration is negative, because it points down.
The Force of gravity
Mass is a measure of an object's inertia. Mass also determines the strength of gravity. Because of gravity all objects are attracted to each other, but we mostly notice the attraction towards the Earth because it is so large and so close.
It's funny, we have a special word just for the direction of gravity, down.
$$F_g=mg $$
\(F_g\) = the force of gravity, weight [N, Newtons, kg m/s²] vector\(m\) = mass [kg]
\(g\) = acceleration of gravity on Earth = 9.8 [m/s²] vector
The force of gravity depends only on the mass of the object because on the surface of the Earth acceleration from gravity is the same for all objects.. If you aren't on the surface of the Earth, the is a different way to calculate gravity.
So why do feathers fall slower than bricks?
The acceleration of 9.8 m/s² on the surface of the Earth is just an approximation. The gravity of earth changes a bit depending on where you are.
Table: Comparative gravities in various cities around the world
| Location | Acceleration in m/s² | Acceleration in ft/s² |
|---|---|---|
| Amsterdam | 9.813 | 32.19 |
| Athens | 9.800 | 32.15 |
| Auckland | 9.799 | 32.15 |
| Bangkok | 9.783 | 32.1 |
| Brussels | 9.811 | 32.19 |
| Buenos Aires | 9.797 | 32.14 |
| Calcutta | 9.788 | 32.11 |
| Cape Town | 9.796 | 32.14 |
| Chicago | 9.803 | 32.16 |
| Copenhagen | 9.815 | 32.2 |
| Frankfurt | 9.810 | 32.19 |
| Havana | 9.788 | 32.11 |
| Helsinki | 9.819 | 32.21 |
| Istanbul | 9.808 | 32.18 |
| Jakarta | 9.781 | 32.09 |
| Kuwait | 9.793 | 32.13 |
| Lisbon | 9.801 | 32.16 |
| London | 9.812 | 32.19 |
| Los Angeles | 9.796 | 32.14 |
| Madrid | 9.800 | 32.15 |
| Manila | 9.784 | 32.1 |
| Mexico City | 9.779 | 32.08 |
| Montréal | 9.789 | 32.12 |
| New York City | 9.802 | 32.16 |
| Nicosia | 9.797 | 32.14 |
| Oslo | 9.819 | 32.21 |
| Ottawa | 9.806 | 32.17 |
| Paris | 9.809 | 32.18 |
| Rio de Janeiro | 9.788 | 32.11 |
| Rome | 9.803 | 32.16 |
| San Francisco | 9.800 | 32.15 |
| Singapore | 9.781 | 32.09 |
| Skopje | 9.804 | 32.17 |
| Stockholm | 9.818 | 32.21 |
| Sydney | 9.797 | 32.14 |
| Taipei | 9.790 | 32.12 |
| Tokyo | 9.798 | 32.15 |
| Vancouver | 9.809 | 32.18 |
| Washington, D.C. | 9.801 | 32.16 |
| Wellington | 9.803 | 32.16 |
| Zurich | 9.807 | 32.18 |
What is the force of gravity in Los Angeles?
Why would the acceleration of gravity seem lower near the equator?
solution
$$F_g=mg$$ $$F_g=(0.44 \, \mathrm{kg} )(9.8\, \mathrm{\tfrac{m}{s^2}})$$ $$F_g=4.3 \, \mathrm{N}$$solution
$$F_g=mg$$ $$\frac{F_g}{g}=m$$ $$\frac{1.8 \,\mathrm{N}}{9.8\, \mathrm{\tfrac{m}{s^2}}}=m$$ $$0.18 \, \mathrm{kg}=m$$Weight and Mass
Another word for the force of gravity is weight. An object on the moon would weigh less than it does on Earth because of the lower gravity, but it would still have the same mass.
Earth's gravity does extend into space, but it decreases with distance. It is about ~90% for astronauts in orbit around the earth, but they don't notice any gravity because they are in a freefall.
Freefall means that you are just letting gravity accelerate you without any opposing forces. To keep from falling we are careful to always counter the force of gravity. This can be done with a parachute, or a jet pack, or just the ground.
solution
The mass doesn't change as an object falls.
Weight just means force of gravity, which also doesn't change in a short freefall
$$F_{g}=mg$$ $$F_{g}=(0.34)(9.8)$$ $$F_{g}=3.33 \, \mathrm{N}$$solution
-
$$\text{weight on the earth}$$ $$F_{g}=mg$$ $$F_{g}=(0.95)(9.8)$$ $$F_{g}=9.31 \, \mathrm{N}$$
-
$$\text{weight on the moon}$$ $$F_{g}=ma$$ $$F_{g}=(0.95)(1.6)$$ $$F_{g}=1.52 \, \mathrm{N}$$
Weight changes on different planets, but mass doesn't change.
What is the mass in kilograms of the book on Earth? On Mars?
solution
$$\text{mass is the same everywhere}$$ $$ 1.2\mathrm{lbs} \frac{1\mathrm{kg}}{2.2\mathrm{lbs}} = 0. \overline{54}\mathrm{kg} $$ $$ m = 0. \overline{54} \, \mathrm{kg} $$What is the weight of the book in Newtons on Earth (a = 9.8 m/s²)?
On Mars (a = 3.711 m/s²)?
solution
-
$$\text{weight on Earth}$$ $$F_{g}=mg$$ $$F_{g}=(0.\overline{54})(9.8)$$ $$F_{g}=5.35\, \mathrm{N}$$
-
$$\text{weight on Mars}$$ $$F_{g}=ma$$ $$F_{g}=(0.\overline{54})(3.711)$$ $$F_{g}=2.02\, \mathrm{N}$$
The Normal Force
Typically a normal force will balance the force of gravity to keep an object from accelerating up or down.
A normal force occurs when two objects are in contact. It is perpendicular to the point of contact. A normal force prevents objects from passing through each other.
solution
$$F_{g}=mg$$ $$F_{g}=(100)(-9.8)$$ $$F_{g}=-980\, \mathrm{N}$$$$\sum F=ma$$ $$F_{g}+F_{N}=ma$$ $$F_{N}=ma-F_{g}$$ $$F_{N}=(100)(2)-(-980)$$ $$F_{N}=1180\, \mathrm{N}$$
solution
In the simple case of a flat horizontal surface with no vertical acceleration the force of gravity will always be equal and opposite to the normal force.
$$F_{g}=mg$$ $$F_{g}=(20)(9.8)$$ $$F_{g}=196\, \mathrm{N}$$$$\sum F=ma$$ $$-F_{g}+F_{N}=ma$$ $$F_{N}=ma+F_{g}$$ $$F_{N}=(20)(0) + 196$$ $$F_{N}=196 \, \mathrm{N}$$
solution
-
$$F_{g}=mg$$ $$F_{g}=(20)(9.8)$$ $$F_{g}=196 \, \mathrm{N}$$
-
Separate the gravity vector into components parallel and perpendicular to the ground. Acceleration is zero in the perpendicular direction.
-
$$\text{perpendicular to ground}$$
$$F_{g\perp}=F_{g}\cos(20)$$ $$F_{g\perp}=(196)\cos(20)$$ $$F_{g\perp}=184 \, \mathrm{N}$$
$$\sum F_{\perp}=ma$$ $$-F_{g\perp}+F_{N}=ma$$ $$F_{N}=ma+F_{g\perp}$$ $$F_{N}=(20)(0) + 184$$ $$F_{N}=184 \, \mathrm{N}$$
-
$$\text{parallel to ground}$$
$$F_{g\parallel}=F_{g}\sin(20)$$ $$F_{g\parallel}=(196)\sin(20)$$ $$F_{g\parallel}=67 \, \mathrm{N}$$
$$\sum F_{\parallel}=ma$$ $$F_{g\parallel}=ma$$ $$(67)=(20)a$$ $$3.35 \, \mathrm{\tfrac{m}{s^{2}}}= a$$
Tension Forces
When a person is walking a dog they are able to apply a force on the dog with a leash. They use the tension on the leash to transfer that force from their hand to the dog.
Tension is the pulling force from a chain, string, or rope. Tension is useful for transferring a force over a distance. In most situations, tension is the same both ends.
solution
$$a = 0.023 \, \mathrm{\tfrac{m}{s^{2}}}$$ $$m = 0.5\, \mathrm{g} = 0.0005 \, \mathrm{kg}$$$$\sum F=ma$$ $$-F_{g} + T = ma$$ $$T = ma + F_{g}$$ $$T = ma + mg$$ $$T = (0.0005\, \mathrm{kg})(0.023\, \mathrm{\tfrac{m}{s^2}}) + (0.0005\, \mathrm{kg}) (9.8\, \mathrm{\tfrac{m}{s^2}})$$ $$T = 0.0049115\, \mathrm{N}$$
solution
The tension force is equal and opposite for the person and crate.
$$\text{crate}$$ $$\sum F=ma$$ $$T = 10 (-0.1)$$ $$T = \color{#d3a}-1 \, \mathrm{N}$$$$\text{person}$$ $$\sum F=ma$$ $$F_{\mathrm{person}} + T = ma$$ $$F_{\mathrm{person}} + {\color{#d3a}1 \, \mathrm{N}} = (100 \, \mathrm{kg})(-0.1\, \mathrm{\tfrac{m}{s^2}})$$ $$F_{\mathrm{person}} + 1\, \mathrm{N} = -10\, \mathrm{N}$$ $$F_{\mathrm{person}} = -11\,\mathrm{N}$$
solution
$$\text{dog: horizontal}$$ $$\sum F=ma$$ $$-T_x + F_{\mathrm{dog}} = 0$$ $$-T_x + 100 \, \mathrm{N} = 0$$ $$T_x = 100 \, \mathrm{N}$$The 100 N is only the x-part of the tension force vector. We can find the total tension force with the Pythagorean theorem. At 45° the x and y parts of the force are the same.
$$T^2 = 100^2+100^2$$ $$T^2 = 20\,000$$ $$T = 141 \, \mathrm{N}$$Assume no friction, no air resistance, Earth gravity, and an inelastic rope.
hint
Both free body diagrams share the same tension and acceleration. Build two equations with Newton's second law and solve a system of equations for T and a. While T and a are the same magnitude they have different directions, so watch the sign of acceleration in particular.
This Khan academy video covers this problem in more depth.
solution
$$\text{20kg box: horizontal}$$ $$\sum F=ma_x$$ $$T = ma_x$$ $$T = \color{#d3a}20a_x$$$$\text{10kg box: vertical}$$ $$\sum F=ma_y$$ $$T - F_g = ma_y$$ $$T - mg = ma_y$$ $$T - (10)(9.8) = 10a_y$$
$${\color{#d3a}20a_x} - 98 = 10a_y$$
$$a_x = -a_y$$
$${\color{#d3a}-20a_y} - 98 = 10a_y$$ $$-98 = 10a_y + 20a_y$$ $$-98 = 30a_y$$ $$-3.2\overline{6} \, \mathrm{\tfrac{m}{s^2}}= a_y$$
solution (treating system as one mass)
$$\text{gravity only pulls on the 10kg part}$$ $$F_g = m g$$ $$F_g = (10 \, \mathrm{kg})(9.8\, \mathrm{\tfrac{m}{s^2}})$$ $$F_g = \color{#d3a}98 \, \mathrm{N}$$$$\text{20 kg+10 kg box(es): vertical}$$ $$\sum F=ma$$ $$F_g = (10\, \mathrm{kg}+20\, \mathrm{kg}) a$$ $$F_g = (30\, \mathrm{kg}) a$$ $$98\, \mathrm{N} = (30\,\mathrm{kg}) a$$ $$-3.2\overline{6} \, \mathrm{\tfrac{m}{s^2}}= a$$


| The Dinaisth Flag | ||
|---|---|---|
| #72DEC2 | #000000 | #FFFFFF |
| Audio | Visual | Research |
The Flag of Dinaisth depicts ehrivevnv's ultraviolet reflection upon the Kanikule ocean below a lightless sky.
It is an old rendition of the character for blue, found in the the Shuowen Jiezi, the character dictionary written by Xu Shen, 100 CE. This particular glyph has probably never been used outside of paleography. I found it to be very beautiful, and the word "blue" has a special meaning in the stories from which my handle "neauoire" comes from.
The neauoire Ambigram avatar is used mostly in forums, as an alternative to the 091450 icon. The design is taken from the complete Neau Ambigram, also found here.
The Carmilla icon is taken from the 1995 cyberpunk magazine with the same name. This avatar is primarily being used on Mastodon and various private channels.
The Orb fractal icon was created in 2007 and has come to represent the Trisight, it is usually visible on the main portal pages of this wiki.
The Neauismetic Crest features various elements found in the iconography of the Neon Hermetists. This icon is used mostly on the maps and charts from that world.
The Lietal icon represents the relationship between the language's Elementary Constructs.
A mug shot created with an old abandoned police sketch software for Macintosh, from the 1980s.
The name of our sailboat, written with the alphabet flags.
Here is the XXIIVV logo written in Pascal.
program XXIIVV; procedure DrawPath (scale, thickness, offx, offy: integer); var arc: Rect; begin PenSize(thickness, thickness); {line1} MoveTo(offx, round(offy + (2 * scale))); LineTo(offx, offy + round(0.5 * scale)); {line2} MoveTo(offx + scale, round(offy + (1.5 * scale))); LineTo(offx + scale, offy + round(0.5 * scale)); {line3} MoveTo(offx + (2 * scale), round(offy + (1.5 * scale))); LineTo(offx + (2 * scale), offy + round(0.5 * scale)); {line4} MoveTo(offx + (3 * scale), round(offy + (0.5 * scale))); LineTo(offx + (3 * scale), offy + round(2 * scale)); LineTo(offx + (4 * scale), offy + round(2 * scale)); LineTo(offx + (4 * scale), offy + round(0 * scale)); LineTo(offx + (5 * scale), offy + round(0 * scale)); LineTo(offx + (5 * scale), offy + round(2 * scale)); LineTo(offx + (6 * scale), offy + round(2 * scale)); LineTo(offx + (6 * scale), offy + round(0 * scale)); LineTo(offx + (7 * scale), offy + round(2 * scale)); LineTo(offx + (8 * scale), offy + round(0 * scale)); LineTo(offx + (9 * scale), offy + round(2 * scale)); { arc1 } SetRect(arc, offx, offy, offx + scale + thickness, offy + scale + thickness); FrameArc(arc, 270, 180); { arc2 } SetRect(arc, offx + (scale), offy + scale, offx + (2 * scale) + thickness, offy + (2 * scale) + thickness); FrameArc(arc, 90, 180); { arc3 } SetRect(arc, offx + (scale * 2), offy, offx + (3 * scale) + thickness, offy + scale + thickness); FrameArc(arc, 270, 180); end; begin { Bresenham 4, 6, 7, 10, 11, 14, 17, 20, 23, 24, 27, 30, 37 } ShowDrawing; DrawPath(27, 1, 20, 25); end.
xiv10.gif | |
xiv12.gif | |
xiv14.gif | |
xiv16.gif |
Here is the XXIIVV logo written in SVG path.


Admit
Invitation
31415926
1
Invite for you
You're
Invited
#1
Date
4/27/2016
Issued By
Ampersand
Invite Number
31415926


Three Kings ***
rated R Warner Bros. Pictures 113 minutes starring George Clooney, Mark Wahlberg, Ice Cube, Spike Jonze, Nora Dunn, Jamie Kennedy, Mykelti Williamson, Cliff Curtis, Said Taghmaoui based on a story by John Ridley written and directed by David O. Russell
I remember critics talking about David O. Russell's first two films, the quirky indies "Spanking The Monkey" and "Flirting With Disaster". The former is about a coming of age story of a boy in the midst of an oedipal relationship, and the latter is about a young man who goes across the country searching for his birth parents. Both are said to be somewhat autobiographical. Both are comedies.
Russell would seem to be an odd choice to write and direct "Three Kings", a story about four Gulf War soldiers who go searching for millions of dollars in gold that Saddam Hussein has stolen from Kuwait. However, despite a number of flaws, David O. Russell's Hollywood debut is a solid, successful effort.
"Three Kings" is not really a war movie. It begins just as the Persian Gulf War has come to an end, and the U.S. army is taking their final prisoners of war. As we learn through title cards which appear on the screen, Archie Gates (George Clooney) is a soldier who's retiring in two weeks, Chief Elgin (Ice Cube) is on an all expenses paid vacation from Detroit, Troy Barlow (Mark Wahlberg) has a wife and young baby at home, and Vig (Spike Jonze, director of the upcoming "Being John Malkovich") would like nothing more than to be Troy Barlow.
The day the war ends, Troy and Vig find a handwritten map while searching an Iraqi war prisoner (I'm not gonna tell you where they find it on him). Upon further inspection by Chief Elgin, the three realize that it is some kind of treasure map, which should lead to the gold which Saddam is believed to have stolen from Kuwait. Archie, who is supposed to be helping out ambitious news reporter Adrianna Cruz (Nora Dunn), gets wind of their little secret, and declares himself the leader.
The four soldiers go AWOL, take a Humvee and jet out across the deserts of Iraq. But when they reach the city which has the gold, they realize they're in for much more than they bargained for. The war is over, but the Iraqi citizens are still in great danger. President Bush has told them to rise up against Saddam, but how are they supposed to do that with no food and few weapons? The soldiers decide to put their greed aside for a while, and "Three Kings" becomes more dramatic than comedic.
The morals of these four men catch up with them, and without thinking, they break the cease fire treaty while trying to save some villagers. Their new mission is to transport almost 100 Iraqi women and children over the border to Iran. However, when Desert Storm hears about this illegal mission, they set out across the desert to find them.
"Three Kings" has some very powerful moments, and it is the first film to portray the Persian Gulf War, where many soldiers were bored by the lack of action. "Three Kings", however, is very, very graphic in its scenes of violence. Some of the shots are done one frame at a time in slow motion, and the bullets are seen numerous times inside a character's stomach. I don't think that was the lesson that filmmakers should have learned from "Saving Private Ryan".
During some parts of the film, I was thankful that it was a comedy. In quite a few scenes, such as when Troy has been captured and is being interrogated by an criminal, it would be very tough to watch if they weren't intercut with a few comedic riffs involving the other 3 soldiers.
The buzz on "Three Kings" has been great, but it's a slick, satisfying film rather than a masterpiece. David O. Russell has done a fine job with this entertaining and self conciously hip movie, but dont expect it to win Oscars like some say it will.
"Three Kings" manages to squeeze out fine performances by George Clooney, Mark Wahlberg and Ice Cube; all three of whom have had their ups and downs. Up and coming actors Spike Jonze and Jamie Kennedy have a couple of annoying little characters, but they each have their own brand of humor and it works.
The ending is a big upper, which should please audiences, but I think that Russell should have followed this powerful film with something more real. Unsung heroes are called unsung heroes because they never are realized for what they are. If these "Three Kings" ended up behind bars for doing something that everyone else was afraid to do, then we would be seeing a more interesting and powerful piece of filmmaking.  Highly recommended, but not for the squeamish.
a review by Akiva Gottlieb<EMAIL_ADDRESS>http://cinemania.8m.com
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Temperature:
Humidity:
Wind speed:
UV Index:
5 Day Forecast:
Temp:
Humidity:
Temp:
Humidity:
Temp:
Humidity:
Temp:
Humidity:
Temp:
Humidity:
Temperature:
Humidity:
Wind speed:
UV Index:
Temp:
Humidity:
Temp:
Humidity:
Temp:
Humidity:
Temp:
Humidity:
Temp:
Humidity:


Performance and Efficiency
- Tez mode
- Timing your UDFs
- Combiner
- Hash-based Aggregation in Map Task
- Memory Management
- Reducer Estimation
- Multi-Query Execution
- Optimization Rules
-
Performance Enhancers
- Use Optimization
- Use Types
- Project Early and Often
- Filter Early and Often
- Reduce Your Operator Pipeline
- Make Your UDFs Algebraic
- Use the Accumulator Interface
- Drop Nulls Before a Join
- Take Advantage of Join Optimizations
- Use the Parallel Features
- Use the LIMIT Operator
- Prefer DISTINCT over GROUP BY/GENERATE
- Compress the Results of Intermediate Jobs
- Combine Small Input Files
- Direct Fetch
- Auto Local Mode
- User Jar Cache
- Specialized Joins
Tez mode
Apache Tez provides an alternative execution engine than MapReduce focusing on performance. By using optimized job flow, edge semantics and container reuse, we see consistent performance boost for both large job and small job.
How to enable Tez
To run Pig in tez mode, simply add "-x tez" in pig command line. Alternatively, you can add "exectype=tez" to conf/pig.properties to change the default exec type to Tez. Java system property "-Dexectype=tez" is also good to trigger the Tez mode.
Prerequisite: Tez requires the tez tarball to be available in hdfs while running a job on the cluster and a tez-site.xml with tez.lib.uris setting pointing to that hdfs location in classpath. Copy the tez tarball to hdfs and add the tez conf directory($TEZ_HOME/conf) containing tez-site.xml to environmental variable "PIG_CLASSPATH" if pig on tez fails with "tez.lib.uris is not defined". This is required by the Apache Pig distribution.
<property> <name>tez.lib.uris</name> <value>${fs.defaultFS}/apps/tez/tez-0.5.2.tar.gz</value> </property>
Tez DAG generation
Every Pig script will be compiled into 1 or more Tez DAG (typically 1). Every Tez DAG consists of a number of vertices and and edges connecting vertices. For example, a simple join involves 1 DAG which consists of 3 vertices: load left input, load right input and join. Do an explain in Tez mode will show you the DAG Pig script compiled into.
Tez session/container reuse
One downside of MapReduce is the startup cost for a job is very high. That hurts the performance especially for small job. Tez alleviate the problem by using session and container reuse, so it is not necessary to start an application master for every job, and start a JVM for every task. By default, session/container reuse is on and we usually shall not turn it off. JVM reuse might cause some side effect if static variable is used since static variable might live across different jobs. So if static variable is used in EvalFunc/LoadFunc/StoreFunc, be sure to implement a cleanup function and register with JVMReuseManager.
Automatic parallelism
Just like MapReduce, if user specify "parallel" in their Pig statement, or user define default_parallel in Tez mode, Pig will honor it (the only exception is if user specify a parallel which is apparently too low, Pig will override it)
If user specify neither "parallel" or "default_parallel", Pig will use automatic parallelism. In MapReduce, Pig submit one MapReduce job a time and before submiting a job, Pig has chance to automatically set reduce parallelism based on the size of input file. On the contrary, Tez submit a DAG as a unit and automatic parallelism is managed in three parts
- Before submiting a DAG, Pig estimate parallelism of each vertex statically based on the input file size of the DAG and the complexity of the pipeline of each vertex
- When DAG progress, Pig adjust the parallelism of vertexes with the best knowledge available at that moment (Pig grace paralellism)
- At runtime, Tez adjust vertex parallelism dynamically based on the input data volume of the vertex. Note currently Tez can only decrease the parallelism dynamically not increase. So in step 1 and 2, Pig overestimate the parallelism
The following parameter control the behavior of automatic parallelism in Tez (share with MapReduce):
pig.exec.reducers.bytes.per.reducer pig.exec.reducers.max
API change
If invoking Pig in Java, there is change in PigStats and PigProgressNotificationListener if using PigRunner.run(), check Pig Statistics and Pig Progress Notification Listener
Known issues
Currently known issue in Tez mode includes:
- Tez local mode is not stable, we see job hang in some cases
- Tez specific GUI is not available yet, there is no GUI to track task progress. However, log message is available in GUI
Timing your UDFs
The first step to improving performance and efficiency is measuring where the time is going. Pig provides a light-weight method for approximately measuring how much time is spent in different user-defined functions (UDFs) and Loaders. Simply set the pig.udf.profile property to true. This will cause new counters to be tracked for all Map-Reduce jobs generated by your script: approx_microsecs measures the approximate amount of time spent in a UDF, and approx_invocations measures the approximate number of times the UDF was invoked. In addition, the frequency of profiling can be configured via the pig.udf.profile.frequency (by default, every 100th invocation). Note that this may produce a large number of counters (two per UDF). Excessive amounts of counters can lead to poor JobTracker performance, so use this feature carefully, and preferably on a test cluster.
Combiner
The Pig combiner is an optimizer that is invoked when the statements in your scripts are arranged in certain ways. The examples below demonstrate when the combiner is used and not used. Whenever possible, make sure the combiner is used as it frequently yields an order of magnitude improvement in performance.
When the Combiner is Used
The combiner is generally used in the case of non-nested foreach where all projections are either expressions on the group column or expressions on algebraic UDFs (see Make Your UDFs Algebraic).
Example:
A = load 'studenttab10k' as (name, age, gpa); B = group A by age; C = foreach B generate ABS(SUM(A.gpa)), COUNT(org.apache.pig.builtin.Distinct(A.name)), (MIN(A.gpa) + MAX(A.gpa))/2, group.age; explain C;
In the above example:
- The GROUP statement can be referred to as a whole or by accessing individual fields (as in the example).
- The GROUP statement and its elements can appear anywhere in the projection.
In the above example, a variety of expressions can be applied to algebraic functions including:
- A column transformation function such as ABS can be applied to an algebraic function SUM.
- An algebraic function (COUNT) can be applied to another algebraic function (Distinct), but only the inner function is computed using the combiner.
- A mathematical expression can be applied to one or more algebraic functions.
You can check if the combiner is used for your query by running EXPLAIN on the FOREACH alias as shown above. You should see the combine section in the MapReduce part of the plan:
..... Combine Plan B: Local Rearrange[tuple]{bytearray}(false) - scope-42 | | | Project[bytearray][0] - scope-43 | |---C: New For Each(false,false,false)[bag] - scope-28 | | | Project[bytearray][0] - scope-29 | | | POUserFunc(org.apache.pig.builtin.SUM$Intermediate)[tuple] - scope-30 | | | |---Project[bag][1] - scope-31 | | | POUserFunc(org.apache.pig.builtin.Distinct$Intermediate)[tuple] - scope-32 | | | |---Project[bag][2] - scope-33 | |---POCombinerPackage[tuple]{bytearray} - scope-36-------- .....
The combiner is also used with a nested foreach as long as the only nested operation used is DISTINCT (see FOREACH and Example: Nested Block).
A = load 'studenttab10k' as (name, age, gpa); B = group A by age; C = foreach B { D = distinct (A.name); generate group, COUNT(D);}
Finally, use of the combiner is influenced by the surrounding environment of the GROUP and FOREACH statements.
When the Combiner is Not Used
The combiner is generally not used if there is any operator that comes between the GROUP and FOREACH statements in the execution plan. Even if the statements are next to each other in your script, the optimizer might rearrange them. In this example, the optimizer will push FILTER above FOREACH which will prevent the use of the combiner:
A = load 'studenttab10k' as (name, age, gpa); B = group A by age; C = foreach B generate group, COUNT (A); D = filter C by group.age <30;
Please note that the script above can be made more efficient by performing filtering before the GROUP statement:
A = load 'studenttab10k' as (name, age, gpa); B = filter A by age <30; C = group B by age; D = foreach C generate group, COUNT (B);
Note: One exception to the above rule is LIMIT. Starting with Pig 0.9, even if LIMIT comes between GROUP and FOREACH, the combiner will still be used. In this example, the optimizer will push LIMIT above FOREACH but this will not prevent the use of the combiner.
A = load 'studenttab10k' as (name, age, gpa); B = group A by age; C = foreach B generate group, COUNT (A); D = limit C 20;
The combiner is also not used in the case where multiple FOREACH statements are associated with the same GROUP:
A = load 'studenttab10k' as (name, age, gpa); B = group A by age; C = foreach B generate group, COUNT (A); D = foreach B generate group, MIN (A.gpa). MAX(A.gpa); .....
Depending on your use case, it might be more efficient (improve performance) to split your script into multiple scripts.
Hash-based Aggregation in Map Task
To improve performance, hash-based aggregation will aggregate records in the map task before sending them to the combiner. This optimization reduces the serializing/deserializing costs of the combiner by sending it fewer records.
Turning On Off
Hash-based aggregation has been shown to improve the speed of group-by operations by up to 50%. However, since this is a very new feature, it is currently turned OFF by default. To turn it ON, set the property pig.exec.mapPartAgg to true.
Configuring
If the group-by keys used for grouping don't result in a sufficient reduction in the number of records, the performance might be worse with this feature turned ON. To prevent this from happening, the feature turns itself off if the reduction in records sent to combiner is not more than a configurable threshold. This threshold can be set using the property pig.exec.mapPartAgg.minReduction. It is set to a default value of 10, which means that the number of records that get sent to the combiner should be reduced by a factor of 10 or more.
Memory Management
Pig allocates a fix amount of memory to store bags and spills to disk as soon as the memory limit is reached. This is very similar to how Hadoop decides when to spill data accumulated by the combiner.
The amount of memory allocated to bags is determined by pig.cachedbag.memusage; the default is set to 20% (0.2) of available memory. Note that this memory is shared across all large bags used by the application.
Reducer Estimation
By default Pig determines the number of reducers to use for a given job based on the size of the input to the map phase. The input data size is divided by the pig.exec.reducers.bytes.per.reducer parameter value (default 1GB) to determine the number of reducers. The maximum number of reducers for a job is limited by the pig.exec.reducers.max parameter (default 999).
The default reducer estimation algorithm described above can be overridden by setting the pig.exec.reducer.estimator parameter to the fully qualified class name of an implementation of org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigReducerEstimator(MapReduce) or org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.TezOperDependencyParallelismEstimator(Tez). The class must exist on the classpath of the process submitting the Pig job. If the pig.exec.reducer.estimator.arg parameter is set, the value will be passed to a constructor of the implementing class that takes a single String.
Multi-Query Execution
With multi-query execution Pig processes an entire script or a batch of statements at once.
Turning it On or Off
Multi-query execution is turned on by default. To turn it off and revert to Pig's "execute-on-dump/store" behavior, use the "-M" or "-no_multiquery" options.
To run script "myscript.pig" without the optimization, execute Pig as follows:
$ pig -M myscript.pig or $ pig -no_multiquery myscript.pig
How it Works
Multi-query execution introduces some changes:
-
For batch mode execution, the entire script is first parsed to determine if intermediate tasks can be combined to reduce the overall amount of work that needs to be done; execution starts only after the parsing is completed (see the EXPLAIN operator and the run and exec commands).
-
Two run scenarios are optimized, as explained below: explicit and implicit splits, and storing intermediate results.
Explicit and Implicit Splits
There might be cases in which you want different processing on separate parts of the same data stream.
Example 1:
A = LOAD ... ... SPLIT A' INTO B IF ..., C IF ... ... STORE B' ... STORE C' ...
Example 2:
A = LOAD ... ... B = FILTER A' ... C = FILTER A' ... ... STORE B' ... STORE C' ...
In prior Pig releases, Example 1 will dump A' to disk and then start jobs for B' and C'. Example 2 will execute all the dependencies of B' and store it and then execute all the dependencies of C' and store it. Both are equivalent, but the performance will be different.
Here's what the multi-query execution does to increase the performance:
-
For Example 2, adds an implicit split to transform the query to Example 1. This eliminates the processing of A' multiple times.
-
Makes the split non-blocking and allows processing to continue. This helps reduce the amount of data that has to be stored right at the split.
-
Allows multiple outputs from a job. This way some results can be stored as a side-effect of the main job. This is also necessary to make the previous item work.
-
Allows multiple split branches to be carried on to the combiner/reducer. This reduces the amount of IO again in the case where multiple branches in the split can benefit from a combiner run.
Storing Intermediate Results
Sometimes it is necessary to store intermediate results.
A = LOAD ... ... STORE A' ... STORE A''
If the script doesn't re-load A' for the processing of A the steps above A' will be duplicated. This is a special case of Example 2 above, so the same steps are recommended. With multi-query execution, the script will process A and dump A' as a side-effect.
Store vs. Dump
With multi-query exection, you want to use STORE to save (persist) your results. You do not want to use DUMP as it will disable multi-query execution and is likely to slow down execution. (If you have included DUMP statements in your scripts for debugging purposes, you should remove them.)
DUMP Example: In this script, because the DUMP command is interactive, the multi-query execution will be disabled and two separate jobs will be created to execute this script. The first job will execute A > B > DUMP while the second job will execute A > B > C > STORE.
A = LOAD 'input' AS (x, y, z); B = FILTER A BY x > 5; DUMP B; C = FOREACH B GENERATE y, z; STORE C INTO 'output';
STORE Example: In this script, multi-query optimization will kick in allowing the entire script to be executed as a single job. Two outputs are produced: output1 and output2.
A = LOAD 'input' AS (x, y, z); B = FILTER A BY x > 5; STORE B INTO 'output1'; C = FOREACH B GENERATE y, z; STORE C INTO 'output2';
Error Handling
With multi-query execution Pig processes an entire script or a batch of statements at once. By default Pig tries to run all the jobs that result from that, regardless of whether some jobs fail during execution. To check which jobs have succeeded or failed use one of these options.
First, Pig logs all successful and failed store commands. Store commands are identified by output path. At the end of execution a summary line indicates success, partial failure or failure of all store commands.
Second, Pig returns different code upon completion for these scenarios:
-
Return code 0: All jobs succeeded
-
Return code 1: Used for retrievable errors
-
Return code 2: All jobs have failed
-
Return code 3: Some jobs have failed
In some cases it might be desirable to fail the entire script upon detecting the first failed job. This can be achieved with the "-F" or "-stop_on_failure" command line flag. If used, Pig will stop execution when the first failed job is detected and discontinue further processing. This also means that file commands that come after a failed store in the script will not be executed (this can be used to create "done" files).
This is how the flag is used:
$ pig -F myscript.pig or $ pig -stop_on_failure myscript.pig
Backward Compatibility
Most existing Pig scripts will produce the same result with or without the multi-query execution. There are cases though where this is not true. Path names and schemes are discussed here.
Any script is parsed in it's entirety before it is sent to execution. Since the current directory can change throughout the script any path used in LOAD or STORE statement is translated to a fully qualified and absolute path.
In map-reduce mode, the following script will load from "hdfs://<host>:<port>/data1" and store into "hdfs://<host>:<port>/tmp/out1".
cd /; A = LOAD 'data1'; cd tmp; STORE A INTO 'out1';
These expanded paths will be passed to any LoadFunc or Slicer implementation. In some cases this can cause problems, especially when a LoadFunc/Slicer is not used to read from a dfs file or path (for example, loading from an SQL database).
Solutions are to either:
-
Specify "-M" or "-no_multiquery" to revert to the old names
-
Specify a custom scheme for the LoadFunc/Slicer
Arguments used in a LOAD statement that have a scheme other than "hdfs" or "file" will not be expanded and passed to the LoadFunc/Slicer unchanged.
In the SQL case, the SQLLoader function is invoked with 'sql://mytable'.
A = LOAD 'sql://mytable' USING SQLLoader();
Implicit Dependencies
If a script has dependencies on the execution order outside of what Pig knows about, execution may fail.
Example
In this script, MYUDF might try to read from out1, a file that A was just stored into. However, Pig does not know that MYUDF depends on the out1 file and might submit the jobs producing the out2 and out1 files at the same time.
... STORE A INTO 'out1'; B = LOAD 'data2'; C = FOREACH B GENERATE MYUDF($0,'out1'); STORE C INTO 'out2';
To make the script work (to ensure that the right execution order is enforced) add the exec statement. The exec statement will trigger the execution of the statements that produce the out1 file.
... STORE A INTO 'out1'; EXEC; B = LOAD 'data2'; C = FOREACH B GENERATE MYUDF($0,'out1'); STORE C INTO 'out2';
Example
In this script, the STORE/LOAD operators have different file paths; however, the LOAD operator depends on the STORE operator.
A = LOAD '/user/xxx/firstinput' USING PigStorage(); B = group .... C = .... agrregation function STORE C INTO '/user/vxj/firstinputtempresult/days1'; .. Atab = LOAD '/user/xxx/secondinput' USING PigStorage(); Btab = group .... Ctab = .... agrregation function STORE Ctab INTO '/user/vxj/secondinputtempresult/days1'; .. E = LOAD '/user/vxj/firstinputtempresult/' USING PigStorage(); F = group .... G = .... aggregation function STORE G INTO '/user/vxj/finalresult1'; Etab =LOAD '/user/vxj/secondinputtempresult/' USING PigStorage(); Ftab = group .... Gtab = .... aggregation function STORE Gtab INTO '/user/vxj/finalresult2';
To make the script work, add the exec statement.
A = LOAD '/user/xxx/firstinput' USING PigStorage(); B = group .... C = .... agrregation function STORE C INTO '/user/vxj/firstinputtempresult/days1'; .. Atab = LOAD '/user/xxx/secondinput' USING PigStorage(); Btab = group .... Ctab = .... agrregation function STORE Ctab INTO '/user/vxj/secondinputtempresult/days1'; EXEC; E = LOAD '/user/vxj/firstinputtempresult/' USING PigStorage(); F = group .... G = .... aggregation function STORE G INTO '/user/vxj/finalresult1'; .. Etab =LOAD '/user/vxj/secondinputtempresult/' USING PigStorage(); Ftab = group .... Gtab = .... aggregation function STORE Gtab INTO '/user/vxj/finalresult2';
If the STORE and LOAD both had exact matching file paths, Pig will recognize the implicit dependency and launch two different mapreduce jobs/Tez DAGs with the second job depending on the output of the first one. exec is not required to be specified in that case.
Optimization Rules
Pig supports various optimization rules, all of which are enabled by default. To disable all or specific optimizations, use one or more of the following methods. Note some optimization rules are mandatory and cannot be disabled.
- The pig.optimizer.rules.disabled pig property, which accepts a comma-separated list of optimization rules to disable; the all keyword disables all non-mandatory optimizations. (e.g.: set pig.optimizer.rules.disabled 'ColumnMapKeyPrune';)
- The -t, -optimizer_off command-line options. (e.g.: pig -optimizer_off [opt_rule | all])
FilterLogicExpressionSimplifier is an exception to the above. The rule is disabled by default, and enabled by setting the pig.exec.filterLogicExpressionSimplifier pig property to true.
PartitionFilterOptimizer
Push the filter condition to loader.
A = LOAD 'input' as (dt, state, event) using HCatLoader(); B = FILTER A BY dt=='201310' AND state=='CA';
Filter condition will be pushed to loader if loader supports (Usually the loader is partition aware, such as HCatLoader)
A = LOAD 'input' as (dt, state, event) using HCatLoader(); --Filter is removed
Loader will be instructed to loader the partition with dt=='201310' and state=='CA'
PredicatePushdownOptimizer
Push the filter condition to loader. Different than PartitionFilterOptimizer, the filter condition will be evaluated in Pig. In other words, the filter condition pushed to the loader is a hint. Loader might still load records which does not satisfy filter condition.
A = LOAD 'input' using OrcStorage(); B = FILTER A BY dt=='201310' AND state=='CA';
Filter condition will be pushed to loader if loader supports
A = LOAD 'input' using OrcStorage(); -- Filter condition push to loader B = FILTER A BY dt=='201310' AND state=='CA'; -- Filter evaluated in Pig again
ConstantCalculator
This rule evaluates constant expression.
1) Constant pre-calculation B = FILTER A BY a0 > 5+7; is simplified to B = FILTER A BY a0 > 12; 2) Evaluate UDF B = FOREACH A generate UPPER(CONCAT('a', 'b')); is simplified to B = FOREACH A generate 'AB';
SplitFilter
Split filter conditions so that we can push filter more aggressively.
A = LOAD 'input1' as (a0, a1); B = LOAD 'input2' as (b0, b1); C = JOIN A by a0, B by b0; D = FILTER C BY a1>0 and b1>0;
Here D will be splitted into:
X = FILTER C BY a1>0; D = FILTER X BY b1>0;
So "a1>0" and "b1>0" can be pushed up individually.
PushUpFilter
The objective of this rule is to push the FILTER operators up the data flow graph. As a result, the number of records that flow through the pipeline is reduced.
A = LOAD 'input'; B = GROUP A BY $0; C = FILTER B BY $0 < 10;
MergeFilter
Merge filter conditions after PushUpFilter rule to decrease the number of filter statements.
PushDownForEachFlatten
The objective of this rule is to reduce the number of records that flow through the pipeline by moving FOREACH operators with a FLATTEN down the data flow graph. In the example shown below, it would be more efficient to move the foreach after the join to reduce the cost of the join operation.
A = LOAD 'input' AS (a, b, c); B = LOAD 'input2' AS (x, y, z); C = FOREACH A GENERATE FLATTEN($0), B, C; D = JOIN C BY $1, B BY $1;
LimitOptimizer
The objective of this rule is to push the LIMIT operator up the data flow graph (or down the tree for database folks). In addition, for top-k (ORDER BY followed by a LIMIT) the LIMIT is pushed into the ORDER BY.
A = LOAD 'input'; B = ORDER A BY $0; C = LIMIT B 10;
ColumnMapKeyPrune
Prune the loader to only load necessary columns. The performance gain is more significant if the corresponding loader support column pruning and only load necessary columns (See LoadPushDown.pushProjection). Otherwise, ColumnMapKeyPrune will insert a ForEach statement right after loader.
A = load 'input' as (a0, a1, a2); B = ORDER A by a0; C = FOREACH B GENERATE a0, a1;
a2 is irrelevant in this query, so we can prune it earlier. The loader in this query is PigStorage and it supports column pruning. So we only load a0 and a1 from the input file.
ColumnMapKeyPrune also prunes unused map keys:
A = load 'input' as (a0:map[]); B = FOREACH A generate a0#'key1';
AddForEach
Prune unused column as soon as possible. In addition to prune the loader in ColumnMapKeyPrune, we can prune a column as soon as it is not used in the rest of the script
-- Original code: A = LOAD 'input' AS (a0, a1, a2); B = ORDER A BY a0; C = FILTER B BY a1>0;
We can only prune a2 from the loader. However, a0 is never used after "ORDER BY". So we can drop a0 right after "ORDER BY" statement.
-- Optimized code: A = LOAD 'input' AS (a0, a1, a2); B = ORDER A BY a0; B1 = FOREACH B GENERATE a1; -- drop a0 C = FILTER B1 BY a1>0;
MergeForEach
The objective of this rule is to merge together two feach statements, if these preconditions are met:
- The foreach statements are consecutive.
- The first foreach statement does not contain flatten.
- The second foreach is not nested.
-- Original code: A = LOAD 'file.txt' AS (a, b, c); B = FOREACH A GENERATE a+b AS u, c-b AS v; C = FOREACH B GENERATE $0+5, v; -- Optimized code: A = LOAD 'file.txt' AS (a, b, c); C = FOREACH A GENERATE a+b+5, c-b;
GroupByConstParallelSetter
Force parallel "1" for "group all" statement. That's because even if we set parallel to N, only 1 reducer will be used in this case and all other reducer produce empty result.
A = LOAD 'input'; B = GROUP A all PARALLEL 10;
Performance Enhancers
Use Optimization
Pig supports various optimization rules which are turned on by default. Become familiar with these rules.
Use Types
If types are not specified in the load statement, Pig assumes the type of =double= for numeric computations. A lot of the time, your data would be much smaller, maybe, integer or long. Specifying the real type will help with speed of arithmetic computation. It has an additional advantage of early error detection.
--Query 1 A = load 'myfile' as (t, u, v); B = foreach A generate t + u; --Query 2 A = load 'myfile' as (t: int, u: int, v); B = foreach A generate t + u;
The second query will run more efficiently than the first. In some of our queries with see 2x speedup.
Project Early and Often
Pig does not (yet) determine when a field is no longer needed and drop the field from the row. For example, say you have a query like:
A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C = join A by t, B by x; D = group C by u; E = foreach D generate group, COUNT($1);
There is no need for v, y, or z to participate in this query. And there is no need to carry both t and x past the join, just one will suffice. Changing the query above to the query below will greatly reduce the amount of data being carried through the map and reduce phases by pig.
A = load 'myfile' as (t, u, v); A1 = foreach A generate t, u; B = load 'myotherfile' as (x, y, z); B1 = foreach B generate x; C = join A1 by t, B1 by x; C1 = foreach C generate t, u; D = group C1 by u; E = foreach D generate group, COUNT($1);
Depending on your data, this can produce significant time savings. In queries similar to the example shown here we have seen total time drop by 50%.
Filter Early and Often
As with early projection, in most cases it is beneficial to apply filters as early as possible to reduce the amount of data flowing through the pipeline.
-- Query 1 A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C = filter A by t == 1; D = join C by t, B by x; E = group D by u; F = foreach E generate group, COUNT($1); -- Query 2 A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C = join A by t, B by x; D = group C by u; E = foreach D generate group, COUNT($1); F = filter E by C.t == 1;
The first query is clearly more efficient than the second one because it reduces the amount of data going into the join.
One case where pushing filters up might not be a good idea is if the cost of applying filter is very high and only a small amount of data is filtered out.
Reduce Your Operator Pipeline
For clarity of your script, you might choose to split your projects into several steps for instance:
A = load 'data' as (in: map[]); -- get key out of the map B = foreach A generate in#'k1' as k1, in#'k2' as k2; -- concatenate the keys C = foreach B generate CONCAT(k1, k2); .......
While the example above is easier to read, you might want to consider combining the two foreach statements to improve your query performance:
A = load 'data' as (in: map[]); -- concatenate the keys from the map B = foreach A generate CONCAT(in#'k1', in#'k2'); ....
The same goes for filters.
Make Your UDFs Algebraic
Queries that can take advantage of the combiner generally ran much faster (sometimes several times faster) than the versions that don't. The latest code significantly improves combiner usage; however, you need to make sure you do your part. If you have a UDF that works on grouped data and is, by nature, algebraic (meaning their computation can be decomposed into multiple steps) make sure you implement it as such. For details on how to write algebraic UDFs, see Algebraic Interface.
A = load 'data' as (x, y, z) B = group A by x; C = foreach B generate group, MyUDF(A); ....
If MyUDF is algebraic, the query will use combiner and run much faster. You can run explain command on your query to make sure that combiner is used.
Use the Accumulator Interface
If your UDF can't be made Algebraic but is able to deal with getting input in chunks rather than all at once, consider implementing the Accumulator interface to reduce the amount of memory used by your script. If your function is Algebraic and can be used on conjunction with Accumulator functions, you will need to implement the Accumulator interface as well as the Algebraic interface. For more information, see Accumulator Interface.
Note: Pig automatically chooses the interface that it expects to provide the best performance: Algebraic > Accumulator > Default.
Drop Nulls Before a Join
With the introduction of nulls, join and cogroup semantics were altered to work with nulls. The semantic for cogrouping with nulls is that nulls from a given input are grouped together, but nulls across inputs are not grouped together. This preserves the semantics of grouping (nulls are collected together from a single input to be passed to aggregate functions like COUNT) and the semantics of join (nulls are not joined across inputs). Since flattening an empty bag results in an empty row (and no output), in a standard join the rows with a null key will always be dropped.
This join
A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C = join A by t, B by x;
is rewritten by Pig to
A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); C1 = cogroup A by t INNER, B by x INNER; C = foreach C1 generate flatten(A), flatten(B);
Since the nulls from A and B won't be collected together, when the nulls are flattened we're guaranteed to have an empty bag, which will result in no output. So the null keys will be dropped. But they will not be dropped until the last possible moment.
If the query is rewritten to
A = load 'myfile' as (t, u, v); B = load 'myotherfile' as (x, y, z); A1 = filter A by t is not null; B1 = filter B by x is not null; C = join A1 by t, B1 by x;
then the nulls will be dropped before the join. Since all null keys go to a single reducer, if your key is null even a small percentage of the time the gain can be significant. In one test where the key was null 7% of the time and the data was spread across 200 reducers, we saw a about a 10x speed up in the query by adding the early filters.
Take Advantage of Join Optimizations
Regular Join Optimizations
Optimization for regular joins ensures that the last table in the join is not brought into memory but streamed through instead. Optimization reduces the amount of memory used which means you can avoid spilling the data and also should be able to scale your query to larger data volumes.
To take advantage of this optimization, make sure that the table with the largest number of tuples per key is the last table in your query. In some of our tests we saw 10x performance improvement as the result of this optimization.
small = load 'small_file' as (t, u, v); large = load 'large_file' as (x, y, z); C = join small by t, large by x;
Specialized Join Optimizations
Optimization can also be achieved using fragment replicate joins, skewed joins, and merge joins. For more information see Specialized Joins.
Use the Parallel Features
You can set the number of reduce tasks for the MapReduce jobs generated by Pig using two parallel features. (The parallel features only affect the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block.)
You Set the Number of Reducers
Use the set default parallel command to set the number of reducers at the script level.
Alternatively, use the PARALLEL clause to set the number of reducers at the operator level. (In a script, the value set via the PARALLEL clause will override any value set via "set default parallel.") You can include the PARALLEL clause with any operator that starts a reduce phase: COGROUP, CROSS, DISTINCT, GROUP, JOIN (inner), JOIN (outer), and ORDER BY. PARALLEL clause can also be used with UNION if Tez is the execution mode. It will turn off the union optimization and introduce an extra reduce step. Though it will have slightly degraded performance due to the extra step, it is very useful for controlling the number of output files.
The number of reducers you need for a particular construct in Pig that forms a MapReduce boundary depends entirely on (1) your data and the number of intermediate keys you are generating in your mappers and (2) the partitioner and distribution of map (combiner) output keys. In the best cases we have seen that a reducer processing about 1 GB of data behaves efficiently.
Let Pig Set the Number of Reducers
If neither "set default parallel" nor the PARALLEL clause are used, Pig sets the number of reducers using a heuristic based on the size of the input data. You can set the values for these properties:
- pig.exec.reducers.bytes.per.reducer - Defines the number of input bytes per reduce; default value is 1000*1000*1000 (1GB).
- pig.exec.reducers.max - Defines the upper bound on the number of reducers; default is 999.
The formula, shown below, is very simple and will improve over time. The computed value takes all inputs within the script into account and applies the computed value to all the jobs within Pig script.
#reducers = MIN (pig.exec.reducers.max, total input size (in bytes) / bytes per reducer)
Examples
In this example PARALLEL is used with the GROUP operator.
A = LOAD 'myfile' AS (t, u, v); B = GROUP A BY t PARALLEL 18; ...
In this example all the MapReduce jobs that get launched use 20 reducers.
SET default_parallel 20; A = LOAD 'myfile.txt' USING PigStorage() AS (t, u, v); B = GROUP A BY t; C = FOREACH B GENERATE group, COUNT(A.t) as mycount; D = ORDER C BY mycount; STORE D INTO 'mysortedcount' USING PigStorage();
Use the LIMIT Operator
Often you are not interested in the entire output but rather a sample or top results. In such cases, using LIMIT can yield a much better performance as we push the limit as high as possible to minimize the amount of data travelling through the pipeline.
Sample:
A = load 'myfile' as (t, u, v); B = limit A 500;
Top results:
A = load 'myfile' as (t, u, v); B = order A by t; C = limit B 500;
Prefer DISTINCT over GROUP BY/GENERATE
To extract unique values from a column in a relation you can use DISTINCT or GROUP BY/GENERATE. DISTINCT is the preferred method; it is faster and more efficient.
Example using GROUP BY - GENERATE:
A = load 'myfile' as (t, u, v); B = foreach A generate u; C = group B by u; D = foreach C generate group as uniquekey; dump D;
Example using DISTINCT:
A = load 'myfile' as (t, u, v); B = foreach A generate u; C = distinct B; dump C;
Compress the Results of Intermediate Jobs
If your Pig script generates a sequence of MapReduce jobs, you can compress the output of the intermediate jobs using LZO compression. (Use the EXPLAIN operator to determine if your script produces multiple MapReduce Jobs.)
By doing this, you will save HDFS space used to store the intermediate data used by PIG and potentially improve query execution speed. In general, the more intermediate data that is generated, the more benefits in storage and speed that result.
You can set the value for these properties:
- pig.tmpfilecompression - Determines if the temporary files should be compressed or not (set to false by default).
- pig.tmpfilecompression.codec - Specifies which compression codec to use. Currently, Pig accepts "gz" and "lzo" as possible values. However, because LZO is under GPL license (and disabled by default) you will need to configure your cluster to use the LZO codec to take advantage of this feature. For details, see http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ.
On the non-trivial queries (one ran longer than a couple of minutes) we saw significant improvements both in terms of query latency and space usage. For some queries we saw up to 96% disk saving and up to 4x query speed up. Of course, the performance characteristics are very much query and data dependent and testing needs to be done to determine gains. We did not see any slowdown in the tests we peformed which means that you are at least saving on space while using compression.
With gzip we saw a better compression (96-99%) but at a cost of 4% slowdown. Thus, we don't recommend using gzip.
Example
-- launch Pig script using lzo compression java -cp $PIG_HOME/pig.jar -Djava.library.path=<path to the lzo library> -Dpig.tmpfilecompression=true -Dpig.tmpfilecompression.codec=lzo org.apache.pig.Main myscript.pig
Combine Small Input Files
Processing input (either user input or intermediate input) from multiple small files can be inefficient because a separate map has to be created for each file. Pig can now combined small files so that they are processed as a single map.
You can set the values for these properties:
- pig.maxCombinedSplitSize – Specifies the size, in bytes, of data to be processed by a single map. Smaller files are combined untill this size is reached.
- pig.splitCombination – Turns combine split files on or off (set to “true” by default).
This feature works with PigStorage. However, if you are using a custom loader, please note the following:
- If your loader implementation makes use of the PigSplit object passed through the prepareToRead method, then you may need to rebuild the loader since the definition of PigSplit has been modified.
- The loader must be stateless across the invocations to the prepareToRead method. That is, the method should reset any internal states that are not affected by the RecordReader argument.
- If a loader implements IndexableLoadFunc, or implements OrderedLoadFunc and CollectableLoadFunc, its input splits won't be subject to possible combinations.
Direct Fetch
When the DUMP operator is used to execute Pig Latin statements, Pig can take the advantage to minimize latency by directly reading data from HDFS rather than launching MapReduce jobs.
The result is fetched if the query contains any of the following operators:
FILTER,
FOREACH,
LIMIT,
STREAM,
UNION.
Fetching will be disabled in case of:
- the presence of other operators, sample loaders and scalar expressions
- no LIMIT operator
- implicit splits
Also note that direct-fetch doesn't support UDFs that interact with the distributed cache. You can check if the query can be fetched by running EXPLAIN. You should see "No MR jobs. Fetch only." in the MapReduce part of the plan.
Direct fetch is turned on by default. To turn it off set the property opt.fetch to false or start Pig with the "-N" or "-no_fetch" option.
Auto Local Mode
Processing small mapreduce jobs on hadoop cluster could be slow as it has overhead of job startup and job scheduling. For jobs with small input data, pig can convert them to run them as in-process mapreduce with hadoop's local mode. If pig.auto.local.enabled flag is set to true, pig will convert mapreduce jobs with input data less than pig.auto.local.input.maxbytes (100MB by default) to run in local mode, provided the number of reducers required by the job are less than or equal to 1. Note, jobs converted to run in local mode load and store data from HDFS, so any job in the pig workflow(dag) could be converted to run in local mode without affecting its downstream jobs.
You can set the values for these properties in order to configure the behavior:
- pig.auto.local.enabled - Turns on/off auto local mode feature (false by default).
- pig.auto.local.input.maxbytes - Controls the max threshold size (in bytes) to convert jobs to run in local mode (100MB by default).
Sometimes, you may want change job configuration for jobs that are converted to run in local mode (eg- change io.sort.mb for small jobs). To do so, you can use pig.local. prefix to any configuration and configuration will be set on converted jobs. For example, set pig.local.io.sort.mb 100 will change io.sort.mb value to 100 for jobs converted to run in local mode.
User Jar Cache
Jars required for user defined functions (UDFs) are copied to distributed cache by pig to make them available on task nodes. To put these jars on distributed cache, pig clients copy these jars to HDFS under a temporary location. For scheduled jobs, these jars do not change frequently. Also, creating a lot of small jar files on HDFS is not HDFS friendly. To avoid copying these small jar files to HDFS again and again, pig allows users to configure a user level jar cache (readable only to the user for security reasons). If pig.user.cache.enabled flag is set to true, UDF jars are copied to jar cache location (configurable) under a directory named with the hash (SHA) of the jar. Hash of the jar is used to identify the existence of the jar in subsequent uses of the jar by the user. If a jar with same hash and filename is found in the cache, it is used avoiding copy of the jar to hdfs.
You can set the values for these properties in order to configure the jar cache:
- pig.user.cache.enabled - Turn on/off user jar cache feature (false by default).
- pig.user.cache.location - Path on HDFS that will be used a staging directory for the user jar cache (defaults to pig.temp.dir or /tmp).
User jar cache feature is fail safe. If jars cannot be copied to jar cache due to any permission/configuration problems, pig will default old behavior.
Specialized Joins
Replicated Joins
Fragment replicate join is a special type of join that works well if one or more relations are small enough to fit into main memory. In such cases, Pig can perform a very efficient join because all of the hadoop work is done on the map side. In this type of join the large relation is followed by one or more small relations. The small relations must be small enough to fit into main memory; if they don't, the process fails and an error is generated.
Usage
Perform a replicated join with the USING clause (see JOIN (inner) and JOIN (outer)). In this example, a large relation is joined with two smaller relations. Note that the large relation comes first followed by the smaller relations; and, all small relations together must fit into main memory, otherwise an error is generated.
big = LOAD 'big_data' AS (b1,b2,b3); tiny = LOAD 'tiny_data' AS (t1,t2,t3); mini = LOAD 'mini_data' AS (m1,m2,m3); C = JOIN big BY b1, tiny BY t1, mini BY m1 USING 'replicated';
Conditions
Fragment replicate joins are experimental; we don't have a strong sense of how small the small relation must be to fit into memory. In our tests with a simple query that involves just a JOIN, a relation of up to 100 M can be used if the process overall gets 1 GB of memory. Please share your observations and experience with us.
In order to avoid replicated joins on large relations, we fail if size of relation(s) to be replicated (in bytes) is greater than pig.join.replicated.max.bytes (default = 1GB).
Bloom Joins
Bloom join is a special type of join where a bloom filter is constructed using join keys of one relation and used to filter records of the other relations before doing a regular hash join. The amount of data sent to the reducers will be a lot less depending up on the numbers of records that are filtered on the map side. Bloom join is very useful in cases where the number of matching records between relations in a join are comparatively less compared to the total records allowing many to be filtered before the join. Before bloom join was added as a type of join, same functionality was achieved by users by using the builtin bloom udfs which is not as efficient and required more lines of code as well. Currently bloom join is only implemented in Tez execution mode. Builtin bloom udfs have to be used for other execution modes.
Usage
Perform a bloom join with the USING clause (see JOIN (inner) and JOIN (outer)). In this example, a large relation is joined with two smaller relations. Note that the large relation comes first followed by the smaller relations. Bloom filter is built from join keys of the right most relation which is small and the filter is applied on the big and medium relations. None of the relations are required to fit into main memory.
big = LOAD 'big_data' AS (b1,b2,b3); medium = LOAD 'medium_data' AS (m1,m2,m3); small = LOAD 'small_data' AS (s1,s2,s3); C = JOIN big BY b1, medium BY m1, small BY s1 USING 'bloom';
In the case of inner join and right outer join, the right most relation is used for building the bloom filter and the users are expected to specify the smaller dataset as the right most relation. But in the case of left outer join, the left most relation is used for building the bloom filter and is expected to be the smaller dataset. This is because all records of the outer relation should be in the result and no records can be filtered. If the left relation turns out to be the bigger dataset, it would not be as efficient to build the bloom filter on the bigger dataset. But it might still perform better than a regular join if it is able to filter lot of records from the right relation.
big = LOAD 'big_data' AS (b1,b2,b3); small = LOAD 'small_data' AS (m1,m2,m3); C = JOIN small BY s1 LEFT, big BY b1 USING 'bloom';
Conditions
- Bloom join cannot be used with a FULL OUTER join.
- If the the underlying data is sufficiently skewed, bloom join might not help. Skewed join can be considered for those cases.
Tuning options
There are multiple pig properties than can be configured to construct a more efficient bloom filter. See Bloom Filter for a discussion of how to select the number of bits and the number of hash functions. Easier option would be to search for "bloom filter calculator" in a search engine and use one of the online bloom filter calculators available to arrive at the desired values.
- pig.bloomjoin.strategy - The valid values for this are 'map' and 'reduce'. Default value is map.
Bloom join has two different kind of implementations to be more efficient in different cases.
In general, there is an extra reduce step in the DAG for construction of the bloom filter(s).
- map - In each map, bloom filters are computed on the join keys partitioned by the hashcode of the key with pig.bloomjoin.num.filters number of partitions. Bloom filters for each partition from different maps are then combined in the reducers producing one bloom filter per partition. The default value of pig.bloomjoin.num.filters is 1 for this strategy and so usually only one bloom filter is created. This is efficient and fast if there are smaller number of maps (<10) and the number of distinct keys are not too high. It can be faster with larger number of maps and even with bigger bloom vector sizes, but the amount of data shuffled to the reducer for aggregation becomes huge making it inefficient.
- reduce - Join keys are sent from the map to the reducer partitioned by hashcode of the key with pig.bloomjoin.num.filters number of partitions. In the reducers, one bloom filter is then computed per partition. Number of reducers are set equal to the number of partitions allowing for each bloom filter to be computed in parallel. The default value of pig.bloomjoin.num.filters is 11 for this strategy. This is efficient for larger datasets with lot of maps or very large bloom vector size. In this case size of keys sent to the reducer is smaller than sending bloom filters to reducer for aggregation making it efficient.
- pig.bloomjoin.num.filters - The number of bloom filters that will be created. Default is 1 for map strategy and 11 for reduce strategy.
- pig.bloomjoin.vectorsize.bytes - The size in bytes of the bit vector to be used for the bloom filter. A bigger vector size will be needed when the number of distinct keys is higher. Default value is 1048576 (1MB).
- pig.bloomjoin.hash.functions - The type of hash function to use. Valid values are 'jenkins' and 'murmur'. Default is murmur.
- pig.bloomjoin.hash.types - The number of hash functions to be used in bloom computation. It determines the probability of false positives. Higher the number lower the false positives. Too high a value can increase the cpu time. Default value is 3.
Skewed Joins
Parallel joins are vulnerable to the presence of skew in the underlying data. If the underlying data is sufficiently skewed, load imbalances will swamp any of the parallelism gains. In order to counteract this problem, skewed join computes a histogram of the key space and uses this data to allocate reducers for a given key. Skewed join does not place a restriction on the size of the input keys. It accomplishes this by splitting the left input on the join predicate and streaming the right input. The left input is sampled to create the histogram.
Skewed join can be used when the underlying data is sufficiently skewed and you need a finer control over the allocation of reducers to counteract the skew. It should also be used when the data associated with a given key is too large to fit in memory.
Usage
Perform a skewed join with the USING clause (see JOIN (inner) and JOIN (outer)).
A = LOAD 'skewed_data' AS (a1,a2,a3); B = LOAD 'data' AS (b1,b2,b3); C = JOIN A BY a1, B BY b1 USING 'skewed';
Conditions
Skewed join will only work under these conditions:
- Skewed join works with two-table inner and outer join. Currently we do not support more than two tables for skewed join. Specifying three-way (or more) joins will fail validation. For such joins, we rely on you to break them up into two-way joins.
- The skewed table must be specified as the left table. Pig samples on that table and determines the number of reducers per key.
- The pig.skewedjoin.reduce.memusage Java parameter specifies the fraction of heap available for the reducer to perform the join. A low fraction forces Pig to use more reducers but increases copying cost. We have seen good performance when we set this value in the range 0.1 - 0.4. However, note that this is hardly an accurate range. Its value depends on the amount of heap available for the operation, the number of columns in the input and the skew. An appropriate value is best obtained by conducting experiments to achieve a good performance. The default value is 0.5.
- Skewed join does not address (balance) uneven data distribution across reducers. However, in most cases, skewed join ensures that the join will finish (however slowly) rather than fail.
Merge Joins
Often user data is stored such that both inputs are already sorted on the join key. In this case, it is possible to join the data in the map phase of a MapReduce job. This provides a significant performance improvement compared to passing all of the data through unneeded sort and shuffle phases.
Pig has implemented a merge join algorithm, or sort-merge join. It works on pre-sorted data, and does not sort data for you. See Conditions, below, for restrictions that apply when using this join algorithm. Pig implements the merge join algorithm by selecting the left input of the join to be the input file for the map phase, and the right input of the join to be the side file. It then samples records from the right input to build an index that contains, for each sampled record, the key(s) the filename and the offset into the file the record begins at. This sampling is done in the first MapReduce job. A second MapReduce job is then initiated, with the left input as its input. Each map uses the index to seek to the appropriate record in the right input and begin doing the join.
Usage
Perform a merge join with the USING clause (see JOIN (inner) and JOIN (outer)).
C = JOIN A BY a1, B BY b1, C BY c1 USING 'merge';
Conditions
Condition A
Inner merge join (between two tables) will only work under these conditions:
- Data must come directly from either a Load or an Order statement.
- There may be filter statements and foreach statements between the sorted data source and the join statement. The foreach statement should meet the following conditions:
- The foreach statement should not change the position of the join keys.
- There should be no transformation on the join keys which will change the sort order.
- UDFs also have to adhere to the previous condition and should not transform the JOIN keys in a way that would change the sort order.
- Data must be sorted on join keys in ascending (ASC) order on both sides.
- If sort is provided by the loader, rather than an explicit Order operation, the right-side loader must implement either the {OrderedLoadFunc} interface or {IndexableLoadFunc} interface.
- Type information must be provided for the join key in the schema.
The PigStorage loader satisfies all of these conditions.
Condition B
Outer merge join (between two tables) and inner merge join (between three or more tables) will only work under these conditions:
- No other operations can be done between the load and join statements.
- Data must be sorted on join keys in ascending (ASC) order on both sides.
- Left-most loader must implement {CollectableLoader} interface as well as {OrderedLoadFunc}.
- All other loaders must implement {IndexableLoadFunc}.
- Type information must be provided for the join key in the schema.
Pig does not provide a loader that supports outer merge joins. You will need to build your own loader to take advantage of this feature.
Merge-Sparse Joins
Merge-Sparse join is a specialization of merge join. Merge-sparse join is intended for use when one of the tables is very sparse, meaning you expect only a small number of records to be matched during the join. In tests this join performed well for cases where less than 1% of the data was matched in the join.
Usage
Perform a merge-sparse join with the USING clause (see JOIN (inner)).
a = load 'sorted_input1' using org.apache.pig.piggybank.storage.IndexedStorage('\t', '0'); b = load 'sorted_input2' using org.apache.pig.piggybank.storage.IndexedStorage('\t', '0'); c = join a by $0, b by $0 using 'merge-sparse'; store c into 'results';
Conditions
Merge-sparse join only works for inner joins and is not currently implemented for outer joins.
For inner joins, the preconditions are the same as for merge join with the exception of constrains on the right-side loader. For sparse-merge joins the loader must implement IndexedLoadFunc or the join will fail.
Piggybank now contains a load function called org.apache.pig.piggybank.storage.IndexedStorage that is a derivation of PigStorage and implements IndexedLoadFunc. This is the only loader included in the standard Pig distribution that can be used for merge-sparse join.
Performance Considerations
Note the following:
- If one of the data sets is small enough to fit into memory, a Replicated Join is very likely to provide better performance.
- You will also see better performance if the data in the left table is partitioned evenly across part files (no significant skew and each part file contains at least one full block of data).


For the fig-swirl: Melt butter over medium heat in a saucepan. Add brown sugar and stir to dissolve. Halve all of the figs and toss in the saucepan with water and lemon juice. Cook over medium heat, stirring frequently, until you have a chunky-jammy mixture. Add salt with one or two stirs, set aside and let cool completely.
Ice cream: In a small pot over medium heat, combine milk, and granulatd sugar until sugar is completely dissolved and the milk is just barely lukewarm. Whisk in the egg yolks. Set mixture in the fridge and wait until the fig mixture is cooled.
Using an ice cream machine, pour liquids into the frozen basin and process according to manufacturer instructions, i.e., let spin and thicken for 20 minutes before adding mascarpone, fig jam mixture, and the nuts. Continue to process for +/- 10 minutes. Pour semi-frozen mixture into a pyrex dish or glass tupperware. Freeze for at least two hours before serving.


Neighbourhood is the best way to stay informed about what’s going on in your
neighborhood—whether it’s finding a last-minute babysitter, planning a local event, or sharing safety
tips. There are so many ways that our neighbors can help us, we just need an easier way to connect with them.


Welina! ‘O Phoenix Maimiti Valentine ko‘u inoa. Ua Hānai ia au o ka Moku ‘o Wai‘anae. I am a student advocate, artist, writer, marathon-er, student filmmaker, and animator, grounded in the Hawaiian culture and the universal message of Peace. To the team I bring a healing balance between members and create a foundation of calmness during our busy days and nights!


{% extends "base.html" %} {% block content %}
{{ book.description }}
| Author: | {{ book.authors.all|join:", " }} |
| ISBN: | {{ book.isbn }} |
| Published: | {{ book.pub_date }} |
| Currently borrowed: | {{ book.borrowing.creation_date|date:"d.m.Y" }} – {{ book.borrowing.due_date|date:"d.m.Y" }} |
| Times borrowed: | {{ book.borrowings_count }} |
| Reservations: | {{ book.reservation__count }} |


R
ahandAmin
Home
About
Our Works
Contact
Home :~
~Biography~
Name:~ Rahand Rizgar M.Amin
Age:~ 19-Nineteen
Knowledge:~ College of Engineering
Job:~ Programmer
Birth City:~ Sulaymanyah-Kurdistan
Skills:~ Programming , Swimming , Musician , Panting
Language:~ Kurdish , English , Turkish