IndianaUniversity  
 
1.Acoordinatedframeworkforconductingdataanalysis

PrinciplesofWorkflowin 2.WFinvolvescoordinatedproceduresfor:
o Planning,organizinganddocumentingresearch

DataAnalysis  o Cleaningdata
o Analyzingdata
   ScottLong o Presentingresults
o Backingupandarchivingmaterials
  



November2010
  

  
1.YourWFmightbe: 1.Replication
A.Plannedandcarefullyorchestrated. o Replicationisessentialforgoodscience.
B.Adhoc,piecemeal,developedinreactiontomistakes. o Aneffectiveworkflowisessentialforreplication.
2.YoucanimproveyourWFwithamodestinvestmentoftime. 2.Gettingtherightanswers
A.Thelessexperienceyouhave,theeasieritis. o Retractionsareembarrassingandcanendcareers.
B.Itwillsaveyoutimeandmakeyouabetterdataanalyst. 3.Time 
  o Scienceisavoraciousinstitution.
o Aneffectiveworkflowmakesyoumoreefficient.
4.Errorsareinevitable;aneffectiveworkflowhelpsyoufindandfixthem.
 

     

5.GainingtheIUadvantage 

1.Easythings:consultingoneasythings,insteadofhardthings.
Thepublicationof[TheWorkflowofDataAnalysis 2.Incorrectresultswithcleverexplanations.
UsingStata]mayevenreduceIndianascomparative
advantageofproducinghotshotquantPhDsnowthat 3.Adissertationdelayed18monthstodeterminewhyresultschanged.
gradstudentselsewherecanvicariouslybenefitfrom
thisimportantaspectofthetrainingthere. Gabriel
Rossmanonhisblog
 4.Irreproducibleresultsfromasingle,743linedofile.
5.Analyzingthewrongdataset:Thedatasetsareexactlythesameexcept
thatIchangedthemarriedvariable.
 
6.AnalyzingthewrongvariablewhilewritinganNASreport.
7.Miscodedgenesthatdelayedprogressinastudyofalcholism.
8.Collaborationsthatmultiplythewaysthingscangowrong.
9.Misleadingorambiguousoutputsuchas...
 

     
Example 1: definitely a problem in a $3M study Example 2: which number is which?
. tab occ ed, row
. tabulate female sdchild_v1
| Years of education
Occupation | 3 6 7 8 9 10 11
R is | Q15 Would let X care for children 12 13 | Total
female? | Defintel Probably Probably Definitel | Total -----------+-------------------------------------------------------------------------------
--------------------+----------
----------+---------------------------------------------+---------- Menial | 0 2 0 0 3 1 3
0Male | 41 99 155 197 | 492 12 2 | 31
1Female | 73 98 156 215 | 542 | 0.00 6.45 0.00 0.00 9.68 3.23 9.68
38.71 6.45 | 100.00
----------+---------------------------------------------+---------- -----------+-------------------------------------------------------------------------------
Total | 114 197 311 412 | 1,034 --------------------+----------

BlueCol | 1 3 1 7 4 6 5
26 7 | 69
  37.68
|
10.14 |
1.45
100.00
4.35 1.45 10.14 5.80 8.70 7.25

-----------+-------------------------------------------------------------------------------
--------------------+----------
Craft | 0 3 2 3 2 2 7
39 7 | 84
| 0.00 3.57 2.38 3.57 2.38 2.38 8.33
46.43 8.33 | 100.00
-----------+-------------------------------------------------------------------------------
--------------------+----------
WhiteCol | 0 0 0 1 0 1 2
19 4 | 41
| 0.00 0.00 0.00 2.44 0.00 2.44 4.88
46.34 9.76 | 100.00
-----------+-------------------------------------------------------------------------------
--------------------+----------
     

Example 3: good software doing things badly  

. logit tenure i.female i.female#c.articles i.male i.male#c.articles, nocons
1.Tacitknowledge  2.Heavylifting  3.Timetopractice
note: 0.male#c.articles omitted because of collinearity
note: 1.male#c.articles omitted because of collinearity 
------------------------------------------------------------------------------
tenure | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.female | -2.473265 .1351561 -18.30 0.000 -2.738166 -2.208364
|
female#|
c.articles |
0 | .0980976 .0098808 9.93 0.000 .0787316 .1174636
1 | .0421485 .0098962 4.26 0.000 .0227524 .0615447
|
1.male | -2.693147 .1170916 -23.00 0.000 -2.922642 -2.463651
|
male#|
c.articles |
0 | (omitted)
1 | (omitted)
------------------------------------------------------------------------------


DidStataCorpreadtheWFbook?   

     

 
1.Explicitknowledgeisthestuffoftextbooksandarticles. Data analysis includes a lot of heavy lifting
 Thereality,ofcourse,todayisthatifyoucomeupwithagreatideayou
2.Tacitknowledgeisimplicitandundocumented(MichaelPolanyi).  don'tgettogoquicklytoasuccessfulproduct.There'salotof
A.Peopleareunawareoftheiressentialtacitknowledge. undifferentiatedheavyliftingthatstandsbetweenyourideaandthat
o HenryBessemerspatentformakingsteeldidntwork(1855) success.JeffBezos,amazon.com

B.Tacitknowledgeistransferredatthebench.  
o Personalcomputersimpedethetransferoftacitknowledge.
 

     
TheWorkflowofDataAnalysisUsingStata  ironicaloptimism
 1.MakestacitknowledgeaboutWFexplicit. Theuniversalaptitudeforineptitudemakesanyhumanaccomplishment
 2.Itdealswithalotofundifferentiatedheavylifting. anincrediblemiracle.Dr.JohnPaulStapp


 3.Itcontainsspecificsonthegeneralissuesdiscussedtoday.
 4.ThebookfocusesontoolsinStata,buttheprinciplesapplybroadly.

  
     

 replication 
1.AneffectiveWFfacilitatesreplication. 1.Thecurseofdimensionality:10minordecisions,leadsto1,024reasonable
waystocreateyourdata.
2.Youmustplanforreplicationatthestartofaproject.
o Wheretotruncateavariable.
3.Disciplinesareincreasinglyconcernedwithreplicability.
o TheseedfortheRNgenerator.
o ArticlesinPoliticalScience,Economics,Sociologyandotherfields. 
o Creatingascalewithpartialmissingdata.
4.Askyourself:
o Whichcasestokeepforanalysis.
o Areyourdofilesandlogfilesreadyforpublicdisplay?
o Howtocodeeducation?
o Willtheyproduceexactlythesameresultsasyouhavepublished?
  o Whatvaluestoassignincomegreaterthan$200,000?
o Andsoon...

 

     

Decisions in the path to analysis: the choices that could be made Decisions in the path to analysis: the choices made

 
     
 Criteria 
2.Documentation:Replicationshouldinvolveretrievingdocumentation,not
tryingtorememberwhatyoudid. 
3.Changingsoftware:2weeksofsleeplessnightsduetoversionvariation.This o Ifyourprogramisnotcorrect,thennothingelsematters.
isparticularlydifficultwhenthereisanactiveusercommunity.           OliveiraandStewart
4.Lostfiles:corrupted,lost,unreadable,obsolete,orambiguousfiles. 

o Completingworkquicklygivenaccuracyandreplicability.
 
o Tensionbetweenworkingquicklyandworkingcarefully.


o Don'trepeatedlyandinconsistentlydecidehowtodothings.
o Standardizationmakesiteasiertofindmistakes.
 

     

 
o Automatedprocedurespreventmistakesandarefaster. o Collaborationmakesitmoredifficulttohaveaneffective,efficientand
o Drukker'sDictim:Nevertypeanythingthatyoucanobtainfromasaved replicableworkflow.
result.(Didtheauthorsofmarginsthinkaboutthis?) o Why?And,whycanttheydoitjustlikeme?
 o Everyproblemyoucanhaveworkingbyyourselfismultipled.
 
o Themorecomplicatedyourproceduresthemorelikelyyouwillmake
mistakesorabandonyourplan.


o Yourworkflowshouldreflectthewayyou liketowork. 
o Ifyouignoreyourprocedures,itisnotagoodWF.


o Differentprojectsrequiredifferentworkflows.
 
     

 

     
 

     

 
1. Agreeduponstandards
2. Explicitcoordination
3. Enforcementofstandards
4. Asenseofhumor
 

     

Steps 
o Incorporateoutputintoyourpresentation.

o Maintaintheprovenanceofresults.
 o Makeeffectivepresentations.
o Datamustbeaccurate. 
o Variablesmustbecarefullynamedandlabeled.
o Backingupandarchiving:preservingthebitsandthecontent.
o Thistakes90%ofthetime,unlessyouhurry.
$2,000toget1variablefromanarchivedfile.
 o Replicationisimpossiblewithoutyourdataanddofiles.
o Estimatemodelsandcreategraphs. o "Today'snoiseistomorrow'sknowledge."DavidClemmer
 
o Oftenthesimplestpartoftheworkflow.
 

     
Tasks 
The ideal
BlauandDuncan(1967)TheAmericanOccupationalStructure 
o Allanalyseswerespecified9monthsbeforeoutputwasreceived.
o Thebookwaswrittenbasedentirelyonthoseanalyses.
o Noneofthelaterbookswrittenwithfullaccesstothedatawereasgood.
 

     

Issues in planning 
1.Aplanisaremindertostayontrack,finishtheproject,andpublishresults.
1.Organizationismovtivatedbytheneedto:
   Work.Finish.Publish.MichaelFaradayssigninhislab
o Findthings
2.Alittleplanninggoesalongwayandalmostalwayssavestime.
o Avoidduplication
3.Planningincludes:
2.Itrequiresexplicit,consistentdecisionsaboutnamingandstoring
o Generalgoals,publishingplans,andfirmdeadlines.
things.
o Divisionoflaborandaccountability.
3.Organization:
o Proposalfordataconstruction:names,labels,formats.
o Helpsyouworkfaster
o Proceduresforhandlingmissingdata.
o Rewardsconsistencyanduniformity
o Anticipatedanalyses.
o Organizationiscontagious
o Guidelinesandresponsibilityfordocumentation.
 
o Proceduresandscheduleforbackingupandarchivingmaterials.

     

 
1.Youcan'tfindafileandthinkyoudeletedit. 1.Itiseasiertocreateafilethantofindafile. 
2.Youfindmultipleversionsofafileanddon'tknowwhichiswhich. 2.Itiseasiertofindafilethantoknowwhatisinthefile.
3.Youandacolleagueareworkingondifferentversionsofthesamepaper. 3.Withdiskspacesocheap,itistemptingtocreatealotoffiles.
Youchangedwhatshechangedandnowyouhavethreeversionsofthe  
paper.
4.Youneedthefinalversionofthepaperthewassubmittedforreview,but
youhavetwo(or16)fileswith"final"inthename.
o final_report_v16.docx
o NSF_science_report20101021.docx
 

     
Organizing: a standard directory structure for all projects Organizing: wfsetupsingle.bat makes it easy
\WF project REM workflow talk 2 \ wfsetupsingle.bat jsl 2009-07-12
REM directory structure for single person.
\- History FOR /F "tokens=2,3,4 delims=/- " %%a in ("%DATE%") do set CDATE=%%c-%%a-%%b
md "- History\%cdate% project directory created"
\2009-03-06 project directory created md "- Hold then delete "
\- Hold then delete md "- Pre posted "
md "- To clean"
\- Pre posted md "Documentation"
md "Posted"
\- To clean md "Resources"
\Documentation md "Text\- Versions\"
md "Work\- To do"
\Posted
\Resources
 
\Text
\- Versions
\Work
\- To do

Forexample,abatchfilemakescreatinguniformdirectorieseasy.
     

 
capture log close
log using wftalk-example, replace text
// program: wftalk-example.do
// task:
// project:
// author: jsl \ 2010-07-27
version 11
clear all
set linesize 80
local tag "wftalk-example.do jsl 2010-07-27"
// #1
// Description of task 1

// #2
// Description of task 2


log close
exit

 Anycoloryouwantaslongasitisblack. 
Templatesmakethisstructureeasytouse.
     

 
1.Long'sLaw:Itisalwaysfastertodocumentittodaythantomorrow.
  Corollary1:Nobodylikestowritedocumentation.
  Corollary2:Nobodyregretshavingwrittendocumentation.
 Haveyoueversaid:"Drat,thisprogramhastoomanycomments."
2.Documentationoccursonmanylevels:logs,metadata,comments,names.
3.Withoutdocumentation,replicationisvirtuallyimpossible,mistakesare
morelikely,andworktakeslonger. 
4.Themorecodifiedthefieldthegreatertheemphasisondocumentation.
 A.TheResearchLogbytheAmericanChemicalSociety.
 B.Lossoftenureforanalteredresearchlog.
 


     
  

1.Doittoday.
2.Checkittomorrowornextweek:italwaysmakessensetoday.
3.Keepupwithdocumentationbytyingittoeventsintheproject.
4.Includefulldatesandnames.



Arealexample... 

     

 Cornell 1975: the entire computing infrastructure



1.Executioninvolvescarryingouttaskswithineachstep.
2.Effectiveexecutionrequirestherighttools.
o Software
a.Texteditor
b.Filemanager
c.Statisticalsoftware
  
d.Macroprogram(evenifonlytoinserttimestamps)
  IBM370with240Kmemory  Winchesterdriveswith3MBstorage
e.Wordprocessor
  
o Hardware:display,storage,memory,CPU

3.Planningisprobablymoreimportantthancomputingpower.
Costofcomputing$1,000,000.

Meantimetodegree7.6years.
For example
   
     

Indiana 2009: a disposable PC 



1.Randomlydivideyourselvesintotwogroups.
o Thecomputerscancomputewhenevertheywantto.
o Theplannerscanonlycomputefortwosixhoursessionsaweek.
2.Whofinishesfirst?

 
   
   Asus1000HEwith2GBmemory   FreeAgentwith1TBstorage
    10,000timesmore      350,000timesmore...

Costofcomputing$400(2,500timesless).
Meantimetodegree7.6years. 

     
Principlesforacomputingworkflow 
1.Dualworkflow:keepdatamanagementanddataanalysisseparate. 
2.Runorder:namefilessothatiftheyarereruninalphabeticalorder,youwill 
produceexactlythesameresults. 
3.Postingprincipleforsharingresults(definedlater)
 Datamanagement==>     
 

              <==Dataanalysis




 

     

Run order and a dual workflow postingprinciple

Datamanagement     Dataanalysis
Thepostingprincipleisdefinedbytworules:
data01.do stat01a.do
data02V2.do stat01b.do 1.Thesharerule:Onlyshareresultsafterthefilesareposted.
data03.do stat01cV2.do 2.Thenochangerule:Onceafileisposted,neverchangeit.
data03-1.do  
data03-2.do stat02a.do
data04.do stat02a1.do
stat02b.do

stat03aV2.do
stat03b.do
stat03c.do
stat03c1.do
stat03c2V2.do
stat03d.do


     

 
1.Lotsofthoughtfulcomments

2.Alignment,indentationandspacing
1.Theyareselfcontained
3.Shortlineswithoutwrapping
2.Theyincludeversioncontrol(version 11.1)
4.Noambiguousabbreviations:  l a l in 1/3
3.Theyexcludedirectoryinformation(whichmightchange)
4.Theyexplicitlysetseedsforrandomnumbers
5.Theyrequirethatyouarchiveuserwrittenadofiles

Simplyput:Itshouldrunonanothercomputeratalaterdatewithoutchanges.
 

     
 
+----------------+
| Key | 1.Muchofdataanalysisinvolvesrepetitivetasks.
|----------------|
| frequency | 2.Repetitioninviteserrors.
| row percentage |
+----------------+
| Years of education
3.Automationisfaster,andlesserrorprone. 

Occupation | 3 6 7 8 9 10
11 12 13 | Total A.macros:wordsthatrepresentstringsoftext.
-----------+---------------------------------------------------------------------------- B.loops:multipleexecutionofthesamecommands.
-----------------------+----------
Menial | 0 2 0 0 3 1 C.returnedresults:avoidingtypingthevalueofanystatisticalresult.
3 12 2 | 31
| 0.00 6.45 0.00 0.00 9.68 3.23
9.68 38.71 6.45 | 100.00
D.matrices:holdandsummarizekeyresults.
-----------+---------------------------------------------------------------------------- E.adofiles:writeprogramsthatdowhatyouwant.
-----------------------+----------
BlueCol | 1 3 1 7 4 6 F.me.hlp:dontkeeplookingupthesamethings.Forexample,
5 26 7 | 69

7.25 37.68
| 1.45
10.14 |
4.35
100.00
1.45 10.14 5.80 8.70 
 
-----------+----------------------------------------------------------------------------
-----------------------+----------
Craft | 0 3 2 3 2 2
7 39 7 | 84
| 0.00 3.57 2.38 3.57 2.38 2.38

     

help me 

 InStata,type:

 findit snag

snagcollectsdozensorhundredsofresultstomakethemeasiertodigest.

o Thestandardoutputisusedtoverifytheresults.
 
o Thesnaggedsummaryletsyoudiscoverwhatyouwant.
o Anyoneusingmarginsknowswhythisisnecessary.

     

 








Example:ownsexandownsexucausedweeksofconfusion.

   
     
 Cleaning 1a: finding an error with a graph
 


 

 

     

Cleaning 1b: reversing the graph Cleaning 2: remembering a coding decision

 

 
   

     

Cleaning 3: understanding the substantive process Cleaning 4: avoiding expensive mistakes

 


 

 

     
 
mlogit (N=337): Factor Change in the Odds of occ

Variable: white (sd=.27642268)

1. Takelotsofclassesinstatistics. Odds comparing

Alternative 1
to Alternative 2 |
|
|
b z P>|z| e^b e^bStdX
------------------+---------------------------------------------
Menial -BlueCol | -1.23650 -1.707 0.088 0.2904 0.7105
Menial -Craft | -0.47234 -0.782 0.434 0.6235 0.8776

2. Findexemplars;dontrediscoverthewheel;dontdoityourway.
Menial -WhiteCol | -1.57139 -1.741 0.082 0.2078 0.6477
Menial -Prof | -1.77431 -2.350 0.019 0.1696 0.6123
BlueCol -Menial | 1.23650 1.707 0.088 3.4436 1.4075
BlueCol -Craft | 0.76416 1.208 0.227 2.1472 1.2352
BlueCol -WhiteCol | -0.33488 -0.359 0.720 0.7154 0.9116
BlueCol -Prof | -0.53780 -0.673 0.501 0.5840 0.8619
Craft -Menial | 0.47234 0.782 0.434 1.6037 1.1395


Craft -BlueCol | -0.76416 -1.208 0.227 0.4657 0.8096
Craft -WhiteCol | -1.09904 -1.343 0.179 0.3332 0.7380
Craft -Prof | -1.30196 -2.011 0.044 0.2720 0.6978
WhiteCol-Menial | 1.57139 1.741 0.082 4.8133 1.5440
WhiteCol-BlueCol | 0.33488 0.359 0.720 1.3978 1.0970
WhiteCol-Craft | 1.09904 1.343 0.179 3.0013 1.3550


WhiteCol-Prof | -0.20292 -0.233 0.815 0.8163 0.9455
Prof -Menial | 1.77431 2.350 0.019 5.8962 1.6331
Prof -BlueCol | 0.53780 0.673 0.501 1.7122 1.1603
Prof -Craft | 1.30196 2.011 0.044 3.6765 1.4332
Prof -WhiteCol | 0.20292 0.233 0.815 1.2250 1.0577
----------------------------------------------------------------

Variable: ed (sd=2.9464271)

Odds comparing |
Alternative 1 |
to Alternative 2 | b z P>|z| e^b e^bStdX

1.Contentandmethodsaresubstantive,disciplinarydecisions.
------------------+---------------------------------------------
Menial -BlueCol | 0.09942 0.972 0.331 1.1045 1.3404
Menial -Craft | -0.09382 -0.962 0.336 0.9105 0.7585
Menial -WhiteCol | -0.35316 -3.011 0.003 0.7025 0.3533
Menial -Prof | -0.77885 -6.795 0.000 0.4589 0.1008
BlueCol -Menial | -0.09942 -0.972 0.331 0.9054 0.7461
BlueCol -Craft | -0.19324 -2.494 0.013 0.8243 0.5659

2.Presentationsandpreservationofprovenanceareuniversal.
BlueCol -WhiteCol | -0.45258 -4.425 0.000 0.6360 0.2636
BlueCol -Prof | -0.87828 -8.735 0.000 0.4155 0.0752
Craft -Menial | 0.09382 0.962 0.336 1.0984 1.3184
Craft -BlueCol | 0.19324 2.494 0.013 1.2132 1.7671
Craft -WhiteCol | -0.25934 -2.773 0.006 0.7716 0.4657
Craft -Prof | -0.68504 -7.671 0.000 0.5041 0.1329
WhiteCol-Menial | 0.35316 3.011 0.003 1.4236 2.8308
WhiteCol-BlueCol | 0.45258 4.425 0.000 1.5724 3.7943
WhiteCol-Craft | 0.25934 2.773 0.006 1.2961 2.1471
WhiteCol-Prof | -0.42569 -4.616 0.000 0.6533 0.2853
Prof -Menial | 0.77885 6.795 0.000 2.1790 9.9228
Prof -BlueCol | 0.87828 8.735 0.000 2.4067 13.3002
Prof -Craft | 0.68504 7.671 0.000 1.9838 7.5264
Prof -WhiteCol | 0.42569 4.616 0.000 1.5307 3.5053
----------------------------------------------------------------

Variable: exper (sd=13.959364)

 

     

 
 

   
   

     

 
ThecircledtextcontainsresultsImayneedtoconfirmlater:



Turningon"show/hide"revealstheprovenance:

 
  
twoway (line art_root2 art_root3 art_root4 art_root5 articles, ///
lwidth(medium)), ytitle(Number of Publications to the k-th Root) ///
yscale(range(0 8.)) legend(pos(11) rows(4) ring(0)) ///
caption(wf7-caption.do \ jsl 2008-04-09, size(vsmall))

     
 
Whenitcomestosavingyourwork,expectthingstogowrong,expectthatyou
willdeletethewrongfileattheworstpossibletime,andexpectahosetobeleft
onintheroomaboveyourcomputer.Ifyouexpecttheworst,youmightbeable
topreventit.



 
     


1.KennedyassassinationonNovember22,1963andthe9/11survey. 
2.508KvolumesinobsoleteformatsatBritishMuseum.2MvideosatIU.
3.NeilArmstrong'swalkonthemoononJuly20,1969,thelostmoontapes,
andPinkFloyd'sDarkSideoftheMoon.









  "afuzzygrayblobwadingthroughaninkwell"      DarkSideoftheMoon

     

 

 


  
 

     
 
 


 


 

     

 
 Dropboxandsimilarservices,enterprisemassstorage,localservers.
1. Installtheprogam

2. Dropfilesintothefolder

1.Sizeperdriveincreasedbyafactorofmorethan300,000.
3. RetrievethemfromanymachinewithDropbox
2.Costpergigabytedecreasedbyafactorof7,000,000.
4. Havesharedfoldersforcollaboration 3.AshoeboxfullofportabledrivescanholdenoughIBMcardstofilla30M
5. Avoidsendingattachmentsevenforonetimefileexchanges cubicfootbuilding;60Mcubicfeetnextmonth.Withcompression
 
 

     

 
1.Slowly,systematically,throughtfully. 1.Therearemanyviableworkflows.
2.Finishthelast5%ofthechange. 2.ThekeyadvantageoftheWFbookisthatitiswrittendown.
3.LikePennandTeller,masterafewcooltricks. 3.AlanAcockwrote: 
4.Don'tdoitunderdeadline. o Noteveryonewillagreewithallof[Long's]suggestions.
  o IwillposttheannouncementofWorkflowonmydoorwiththefollowing
note:Iamgladtohelpanybodywhofollowedatleast25%oftheadvice
Longprovidesandbringsmetheirdofiles! 
4.DoyoureallywanttospendyourtimerediscoveringthemistakesImade?