Everand Logo

Welcome to Everand!

  • Discover millions of audiobooks, ebooks, and so much more with a free trial

    From $11.99/month after trial. Cancel anytime.

    Parallel Programming with Python
    Parallel Programming with Python
    Parallel Programming with Python
    Ebook230 pages1 hour

    Parallel Programming with Python

    By Jan Palach

    ()

    Read preview

    About this ebook

    A fast, easy-to-follow and clear tutorial to help you develop Parallel computing systems using Python. Along with explaining the fundamentals, the book will also introduce you to slightly advanced concepts and will help you in implementing these techniques in the real world.
    If you are an experienced Python programmer and are willing to utilize the available computing resources by parallelizing applications in a simple way, then this book is for you. You are required to have a basic knowledge of Python development to get the most of this book.
    LanguageEnglish
    PublisherPackt Publishing
    Release dateJun 25, 2014
    ISBN9781783288403
    Parallel Programming with Python

    Related to Parallel Programming with Python

    Related ebooks

    Programming For You

    View More
    View More

    Related podcast episodes

    Related categories

    Reviews for Parallel Programming with Python

    0 ratings

    0 ratings0 reviews

    What did you think?

    Tap to rate

    Review must be at least 10 words

      Book preview

      Parallel Programming with Python - Jan Palach

      Table of Contents

      Parallel Programming with Python

      Credits

      About the Author

      Acknowledgments

      About the Reviewers

      www.PacktPub.com

      Support files, eBooks, discount offers, and more

      Why subscribe?

      Free access for Packt account holders

      Preface

      What this book covers

      What you need for this book

      Who this book is for

      Conventions

      Reader feedback

      Customer support

      Downloading the example code

      Errata

      Piracy

      Questions

      1. Contextualizing Parallel, Concurrent, and Distributed Programming

      Why use parallel programming?

      Exploring common forms of parallelization

      Communicating in parallel programming

      Understanding shared state

      Understanding message passing

      Identifying parallel programming problems

      Deadlock

      Starvation

      Race conditions

      Discovering Python's parallel programming tools

      The Python threading module

      The Python multiprocessing module

      The parallel Python module

      Celery – a distributed task queue

      Taking care of Python GIL

      Summary

      2. Designing Parallel Algorithms

      The divide and conquer technique

      Using data decomposition

      Decomposing tasks with pipeline

      Processing and mapping

      Identifying independent tasks

      Identifying the tasks that require data exchange

      Load balance

      Summary

      3. Identifying a Parallelizable Problem

      Obtaining the highest Fibonacci value for multiple inputs

      Crawling the Web

      Summary

      4. Using the threading and concurrent.futures Modules

      Defining threads

      Advantages and disadvantages of using threads

      Understanding different kinds of threads

      Defining the states of a thread

      Choosing between threading and _thread

      Using threading to obtain the Fibonacci series term with multiple inputs

      Crawling the Web using the concurrent.futures module

      Summary

      5. Using Multiprocessing and ProcessPoolExecutor

      Understanding the concept of a process

      Understanding the process model

      Defining the states of a process

      Implementing multiprocessing communication

      Using multiprocessing.Pipe

      Understanding multiprocessing.Queue

      Using multiprocessing to compute Fibonacci series terms with multiple inputs

      Crawling the Web using ProcessPoolExecutor

      Summary

      6. Utilizing Parallel Python

      Understanding interprocess communication

      Exploring named pipes

      Using named pipes with Python

      Writing in a named pipe

      Reading named pipes

      Discovering PP

      Using PP to calculate the Fibonacci series term on SMP architecture

      Using PP to make a distributed Web crawler

      Summary

      7. Distributing Tasks with Celery

      Understanding Celery

      Why use Celery?

      Understanding Celery's architecture

      Working with tasks

      Discovering message transport (broker)

      Understanding workers

      Understanding result backends

      Setting up the environment

      Setting up the client machine

      Setting up the server machine

      Dispatching a simple task

      Using Celery to obtain a Fibonacci series term

      Defining queues by task types

      Using Celery to make a distributed Web crawler

      Summary

      8. Doing Things Asynchronously

      Understanding blocking, nonblocking, and asynchronous operations

      Understanding blocking operations

      Understanding nonblocking operations

      Understanding asynchronous operations

      Understanding event loop

      Polling functions

      Using event loops

      Using asyncio

      Understanding coroutines and futures

      Using coroutine and asyncio.Future

      Using asyncio.Task

      Using an incompatible library with asyncio

      Summary

      Index

      Parallel Programming with Python

      Parallel Programming with Python

      Copyright © 2014 Packt Publishing

      All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

      Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

      Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

      First published: June 2014

      Production reference: 1180614

      Published by Packt Publishing Ltd.

      Livery Place

      35 Livery Street

      Birmingham B3 2PB, UK.

      ISBN 978-1-78328-839-7

      www.packtpub.com

      Cover image by Lis Marie Martini (<[email protected]>)

      Credits

      Author

      Jan Palach

      Reviewers

      Cyrus Dasadia

      Wei Di

      Michael Galloy

      Ludovic Gasc

      Kamran Hussain

      Bruno Torres

      Commissioning Editor

      Rebecca Youé

      Acquisition Editor

      Llewellyn Rozario

      Content Development Editor

      Sankalp Pawar

      Technical Editors

      Novina Kewalramani

      Humera Shaikh

      Copy Editors

      Roshni Banerjee

      Sarang Chari

      Gladson Monteiro

      Project Coordinator

      Lima Danti

      Proofreaders

      Simran Bhogal

      Maria Gould

      Paul Hindle

      Indexers

      Mehreen Deshmukh

      Rekha Nair

      Tejal Soni

      Priya Subramani

      Graphics

      Disha Haria

      Abhinash Sahu

      Production Coordinator

      Saiprasad Kadam

      Cover Work

      Saiprasad Kadam

      About the Author

      Jan Palach has been a software developer for 13 years, having worked with scientific visualization and backend for private companies using C++, Java, and Python technologies. Jan has a degree in Information Systems from Estácio de Sá University, Rio de Janeiro, Brazil, and a postgraduate degree in Software Development from Paraná State Federal Technological University. Currently, he works as a senior system analyst at a private company within the telecommunication sector implementing C++ systems; however, he likes to have fun experimenting with Python and Erlang—his two technological passions. Naturally curious, he loves challenges and learning new technologies, meeting new people, and learning about different cultures.

      Acknowledgments

      I had no idea how hard it could be to write a book with such a tight deadline among so many other things taking place in my life. I had to fit the writing into my routine, taking care of my family, karate lessons, work, Diablo III, and so on. The task was not easy; however, I got to the end of it hoping that I have generated quality content to please most readers, considering that I have focused on the most important thing based on my experience.

      The list of people I would like to acknowledge is so long that I would need a book only for this. So, I would like to thank some people I have constant contact with and who, in a direct or indirect way, helped me throughout this quest.

      My wife Anicieli Valeska de Miranda Pertile, the woman I chose to share my love with and gather toothbrushes with to the end of this life, who allowed me to have the time to create this book and did not let me give up when I thought I could not make it. My family has always been important to me during my growth as a human being and taught me the path of goodness.

      I would like to thank Fanthiane Ketrin Wentz, who beyond being my best friend is also guiding me through the ways of martial arts, teaching me the values I will carry during a lifetime—a role model for me. Lis Marie Martini, dear friend who provided the cover for this book, and who is an incredible photographer and animal lover.

      Big thanks to my former English teacher, reviser, and proofreader, Marina Melo, who helped along the writing of this book. Thanks to the reviewers and personal friends, Vitor Mazzi and Bruno Torres, who contributed a lot to my professional growth and still do.

      Special thanks to Rodrigo Cacilhas, Bruno Bemfica, Rodrigo Delduca, Luiz Shigunov, Bruno Almeida Santos, Paulo Tesch (corujito), Luciano Palma, Felipe Cruz, and other people with whom I often talk to about technology. A special thanks to Turma B.

      Big thanks to Guido Van Rossum for creating Python, which transformed programming into something pleasant; we need more of this stuff and less set/get.

      About the Reviewers

      Cyrus Dasadia has worked as a Linux system administrator for over a decade for organizations such as AOL and InMobi. He is currently developing CitoEngine, an open source alert management service written entirely in Python.

      Wei Di is a research scientist at eBay Research Labs, focusing on advanced computer vision, data mining, and information retrieval technologies for large-scale e-commerce applications. Her interest covers large-scale data mining, machine learning in merchandising, data quality for e-commerce, search relevance, and ranking and recommender systems. She also has years of research experience in pattern recognition and image processing. She received her PhD from Purdue University in 2011 with focuses on data mining and image classification.

      Michael Galloy works as a research mathematician for Tech-X Corporation involved in scientific visualizations using IDL and Python. Before that, he worked for five years teaching all levels of IDL programming and consulting for Research Systems, Inc. (now Exelis Visual Information Solutions). He is the author of Modern IDL (modernidl.idldev.com) and is the creator/maintainer

      Enjoying the preview?
      Page 1 of 1