book

Python for Data Analysis

by Wes McKinney

October 2012

Beginner to intermediate

463 pages

12h 53m

English

O'Reilly Media, Inc.

Start your free trial

Related skills

Data Science

Associated roles

Business analyst
Data architect
Data scientist
Python developer

Conventions Used in This BookUsing Code ExamplesSafari® Books OnlineHow to Contact Us
What Is This Book About?Why Python for Data Analysis?Python as GlueSolving the “Two-Language” ProblemWhy Not Python?Essential Python LibrariesNumPypandasmatplotlibIPythonSciPyInstallation and SetupWindowsApple OS XGNU/LinuxPython 2 and Python 3Integrated Development Environments (IDEs)Community and ConferencesNavigating This BookCode ExamplesData for ExamplesImport ConventionsJargonAcknowledgements
1.usa.gov data from bit.lyCounting Time Zones in Pure PythonCounting Time Zones with pandasMovieLens 1M Data SetMeasuring rating disagreementUS Baby Names 1880-2010Analyzing Naming TrendsMeasuring the increase in naming diversityThe “Last letter” RevolutionBoy names that became girl names (and vice versa)Conclusions and The Path Ahead
IPython BasicsTab CompletionIntrospectionThe %run CommandInterrupting running codeExecuting Code from the ClipboardIPython interaction with editors and IDEsKeyboard ShortcutsExceptions and TracebacksMagic CommandsQt-based Rich GUI ConsoleMatplotlib Integration and Pylab ModeUsing the Command HistorySearching and Reusing the Command HistoryInput and Output VariablesLogging the Input and OutputInteracting with the Operating SystemShell Commands and AliasesDirectory Bookmark SystemSoftware Development ToolsInteractive DebuggerOther ways to make use of the debuggerTiming Code: %time and %timeitBasic Profiling: %prun and %run -pProfiling a Function Line-by-LineIPython HTML NotebookTips for Productive Code Development Using IPythonReloading Module DependenciesCode Design TipsKeep relevant objects and data aliveFlat is better than nestedOvercome a fear of longer filesAdvanced IPython FeaturesMaking Your Own Classes IPython-friendlyProfiles and ConfigurationCredits
The NumPy ndarray: A Multidimensional Array ObjectCreating ndarraysData Types for ndarraysOperations between Arrays and ScalarsBasic Indexing and SlicingIndexing with slicesBoolean IndexingFancy IndexingTransposing Arrays and Swapping AxesUniversal Functions: Fast Element-wise Array FunctionsData Processing Using ArraysExpressing Conditional Logic as Array OperationsMathematical and Statistical MethodsMethods for Boolean ArraysSortingUnique and Other Set LogicFile Input and Output with ArraysStoring Arrays on Disk in Binary FormatSaving and Loading Text FilesLinear AlgebraRandom Number GenerationExample: Random WalksSimulating Many Random Walks at Once
Introduction to pandas Data StructuresSeriesDataFrameIndex ObjectsEssential FunctionalityReindexingDropping entries from an axisIndexing, selection, and filteringArithmetic and data alignmentArithmetic methods with fill valuesOperations between DataFrame and SeriesFunction application and mappingSorting and rankingAxis indexes with duplicate valuesSummarizing and Computing Descriptive StatisticsCorrelation and CovarianceUnique Values, Value Counts, and MembershipHandling Missing DataFiltering Out Missing DataFilling in Missing DataHierarchical IndexingReordering and Sorting LevelsSummary Statistics by LevelUsing a DataFrame’s ColumnsOther pandas TopicsInteger IndexingPanel Data
Reading and Writing Data in Text FormatReading Text Files in PiecesWriting Data Out to Text FormatManually Working with Delimited FormatsJSON DataXML and HTML: Web ScrapingParsing XML with lxml.objectifyBinary Data FormatsUsing HDF5 FormatReading Microsoft Excel FilesInteracting with HTML and Web APIsInteracting with DatabasesStoring and Loading Data in MongoDB
Combining and Merging Data SetsDatabase-style DataFrame MergesMerging on IndexConcatenating Along an AxisCombining Data with OverlapReshaping and PivotingReshaping with Hierarchical IndexingPivoting “long” to “wide” FormatData TransformationRemoving DuplicatesTransforming Data Using a Function or MappingReplacing ValuesRenaming Axis IndexesDiscretization and BinningDetecting and Filtering OutliersPermutation and Random SamplingComputing Indicator/Dummy VariablesString ManipulationString Object MethodsRegular expressionsVectorized string functions in pandasExample: USDA Food Database

A Brief matplotlib API PrimerFigures and SubplotsAdjusting the spacing around subplotsColors, Markers, and Line StylesTicks, Labels, and LegendsSetting the title, axis labels, ticks, and ticklabelsAdding legendsAnnotations and Drawing on a SubplotSaving Plots to Filematplotlib ConfigurationPlotting Functions in pandasLine PlotsBar PlotsHistograms and Density PlotsScatter PlotsPlotting Maps: Visualizing Haiti Earthquake Crisis DataPython Visualization Tool EcosystemChacomayaviOther PackagesThe Future of Visualization Tools?
GroupBy MechanicsIterating Over GroupsSelecting a Column or Subset of ColumnsGrouping with Dicts and SeriesGrouping with FunctionsGrouping by Index LevelsData AggregationColumn-wise and Multiple Function ApplicationReturning Aggregated Data in “unindexed” FormGroup-wise Operations and TransformationsApply: General split-apply-combineSuppressing the group keysQuantile and Bucket AnalysisExample: Filling Missing Values with Group-specific ValuesExample: Random Sampling and PermutationExample: Group Weighted Average and CorrelationExample: Group-wise Linear RegressionPivot Tables and Cross-TabulationCross-Tabulations: CrosstabExample: 2012 Federal Election Commission DatabaseDonation Statistics by Occupation and EmployerBucketing Donation AmountsDonation Statistics by State
Date and Time Data Types and ToolsConverting between string and datetimeTime Series BasicsIndexing, Selection, SubsettingTime Series with Duplicate IndicesDate Ranges, Frequencies, and ShiftingGenerating Date RangesFrequencies and Date OffsetsWeek of month datesShifting (Leading and Lagging) DataShifting dates with offsetsTime Zone HandlingLocalization and ConversionOperations with Time Zone−aware Timestamp ObjectsOperations between Different Time ZonesPeriods and Period ArithmeticPeriod Frequency ConversionQuarterly Period FrequenciesConverting Timestamps to Periods (and Back)Creating a PeriodIndex from ArraysResampling and Frequency ConversionDownsamplingOpen-High-Low-Close (OHLC) resamplingResampling with GroupByUpsampling and InterpolationResampling with PeriodsTime Series PlottingMoving Window FunctionsExponentially-weighted functionsBinary Moving Window FunctionsUser-Defined Moving Window FunctionsPerformance and Memory Usage Notes
Data Munging TopicsTime Series and Cross-Section AlignmentOperations with Time Series of Different FrequenciesUsing periods instead of timestampsTime of Day and “as of” Data SelectionSplicing Together Data SourcesReturn Indexes and Cumulative ReturnsGroup Transforms and AnalysisGroup Factor ExposuresDecile and Quartile AnalysisMore Example ApplicationsSignal Frontier AnalysisFuture Contract RollingRolling Correlation and Linear Regression
ndarray Object InternalsNumPy dtype HierarchyAdvanced Array ManipulationReshaping ArraysC versus Fortran OrderConcatenating and Splitting ArraysStacking helpers: r_ and c_Repeating Elements: Tile and RepeatFancy Indexing Equivalents: Take and PutBroadcastingBroadcasting Over Other AxesSetting Array Values by BroadcastingAdvanced ufunc Usageufunc Instance MethodsCustom ufuncsStructured and Record ArraysNested dtypes and Multidimensional FieldsWhy Use Structured Arrays?Structured Array Manipulations: numpy.lib.recfunctionsMore About SortingIndirect Sorts: argsort and lexsortAlternate Sort Algorithmsnumpy.searchsorted: Finding elements in a Sorted ArrayNumPy Matrix ClassAdvanced Array Input and OutputMemory-mapped FilesHDF5 and Other Array Storage OptionsPerformance TipsThe Importance of Contiguous MemoryOther Speed Options: Cython, f2py, C
The Python InterpreterThe BasicsLanguage SemanticsIndentation, not bracesEverything is an objectCommentsFunction and object method callsVariables and pass-by-referenceDynamic references, strong typesAttributes and methods“Duck” typingImportsBinary operators and comparisonsStrictness versus lazinessMutable and immutable objectsScalar TypesNumeric typesStringsBooleansType castingNoneDates and timesControl Flowif, elif, and elsefor loopswhile loopspassException handlingrange and xrangeTernary ExpressionsData Structures and SequencesTupleUnpacking tuplesTuple methodsListAdding and removing elementsConcatenating and combining listsSortingBinary search and maintaining a sorted listSlicingBuilt-in Sequence FunctionsenumeratesortedzipreversedDictCreating dicts from sequencesDefault valuesValid dict key typesSetList, Set, and Dict ComprehensionsNested list comprehensionsFunctionsNamespaces, Scope, and Local FunctionsReturning Multiple ValuesFunctions Are ObjectsAnonymous (lambda) FunctionsClosures: Functions that Return FunctionsExtended Call Syntax with *args, **kwargsCurrying: Partial Argument ApplicationGeneratorsGenerator expresssionsitertools moduleFiles and the operating system

Content preview from Python for Data Analysis

Chapter 4. NumPy Basics: Arrays and Vectorized Computation

NumPy, short for Numerical Python, is the fundamental package required for high performance scientific computing and data analysis. It is the foundation on which nearly all of the higher-level tools in this book are built. Here are some of the things it provides:

ndarray, a fast and space-efficient multidimensional array providing vectorized arithmetic operations and sophisticated broadcasting capabilities
Standard mathematical functions for fast operations on entire arrays of data without having to write loops
Tools for reading / writing array data to disk and working with memory-mapped files
Linear algebra, random number generation, and Fourier transform capabilities
Tools for integrating code written in C, C++, and Fortran

The last bullet point is also one of the most important ones from an ecosystem point of view. Because NumPy provides an easy-to-use C API, it is very easy to pass data to external libraries written in a low-level language and also for external libraries to return data to Python as NumPy arrays. This feature has made Python a language of choice for wrapping legacy C/C++/Fortran codebases and giving them a dynamic and easy-to-use interface.

While NumPy by itself does not provide very much high-level data analytical functionality, having an understanding of NumPy arrays and array-oriented computing will help you use tools like pandas much more effectively. If you’re new to Python and just looking to get your ...