InterAction - Rockefeller - TCC - GlobalGiving
Download as PPTX, PDF0 likes340 views
The session provided an overview of evaluating networks and coalitions. It presented the work of two of the Rockefeller Foundation's grantees: GlobalGiving and their storytelling evaluation work, and evaluating advocacy initiatives and coalitions. Key aspects of evaluation discussed included measuring member and coalition capacity, capturing externalities and cross-fertilization, and developing appropriate outcome questions for inter-organizational relationships (IORs) in different contexts like emergency settings.
InterAction - Rockefeller - TCC - GlobalGiving
- 1. Advances in Development Evaluation InterAction Forum 2012
- 2. Session Overview • Interactive - audience participation • Global Giving’s storytelling work • Evaluation of networks • Collaborative learning activity • Wrap up / Q&A
- 3. Introduction • The Rockefeller Foundation Evaluation Office is supporting innovative evaluation approaches, new methods and tools and evaluation capacity development • This session presents the work of two of RF’s grantees who are key contributors to advances in development evaluation
- 4. Tell us about something that surprised you at InterAction yesterday. SKILLS OTHER THE IDEAS EXPERIENCE What is your story about? Place it somewhere inside the triangle, or in the “other” box.
- 7. GlobalGiving Impact Framework Catalyze Marketplace of $, Info, Ideas Accessible to Growing # of More Effective Orgs Democratizes Aid and Philanthropy
- 8. Design Model: Yelp! Changes the Market
- 9. Closing the Broken Feedback Loop Feedback makes it safe (and satisfying) to give to riskier (but maybe better performing) orgs. Feedback identifies great NGOs and helps them learn. Resulting data is useful for other funders, NGOs, government agencies, and researchers.
- 11. Methodological Logic Story Capture Many Emergence Self-Signification Act on Insights Narratives Contextualized Statistics Visualization Discussion (Theme, Geo, Org, Tim e)
- 12. What’s working? • Real-Time Decision-Making Aid for GlobalGiving Working! • Story Patterns Illuminate Needs • Actionable Constituent Voice • Performance Dashboard for NGOs Promising! • Methodology Recognized as Useful • Interoperability with other Monitoring Tools • Globally Scalable Collection Infrastructure Working on it! • Market-Driven Incentives for Story Collection • Open-Source Analysis Tools
- 13. Evaluating Coalitions and Networks Jared Raynor Director of Evaluation InterAction Conference May 2012 strategies to achieve social impact
- 15. Logic Model: Advocacy Initiatives Acting Project Inputs/ Strategies: Outcomes: Impact: Organization: Resources: Advocacy Staff: Issue analysis/ •Time, experience and research Improvement in the Quality of Living for the Community expertise Better defined and Change in Social Structure Reflecting Positive Social Justice Shift (administrative, legislativ Nonprofit or Advocacy framed problems coalition as a strategy) Media advocacy/ Organization (uses e, election-related and Public awareness legal) raising •Core skills (analytical, communicatio Issues more clearly set n, research, etc.) Grass roots on the public agenda organizing and gain prominence Funding: (momentum, interest, a •Monetary resources Coalition building/ wareness, etc.) •Non-monetary/ non-staff networking resources (e.g. volunteers, in-kind Policy makers adopt donations) Policy analysis/ new policies based on Coalition/Network research the agenda / do not (as an Actor as compared to a Organizational adopt harmful policies strategy) Reputation: Legal action •Advocacy related •General perception as a Lobbying and direct Policy decisions quality organization policy-maker effectively •Representative for base influence implemented constituency Assure good policy Network: implementation •Policy makers (Administrative/ Impact of new policies •Partner organizations Regulatory is evaluated •Media oversight, TA, monit •Mobilized base oring, etc.) (C) TCC Group 2010
- 16. Continuum of Inter-Organizational Relationships (IORs) Mutual Merger Accountability Joint Venture Strategic Alliance Collaboration Formal Informal Organization Organization Coalition Formal Network Committee Separate Accountability Informal Network Adapted from: Wolf, T. (2003). A Practical Approach to Evaluation of Collaborations. In T.E. Backer (Ed.). Evaluating Community Collaborations. New York, NY. Springer Publishing Co. with additional acknowledgement of K. Hobson and M. Hightower King at AEA 2009.
- 17. Relationships for What Purpose? • Respond to complexity • Cope with turbulence/complexity • Acquire resources (while maintaining autonomy) • Decrease transaction costs • Gain legitimacy/power Positive Negative Externalities Externalities (Value) (Congestion)
- 18. What Are We Measuring? Coalition Organization Individual
- 19. An Entanglement of Relations Coalition Organization Organization Number of Individual Relationships Organization 2 Person Coalition: 8 Individual Organization 3 Person Coalition: 18 4 Person Coalition: 32 Individual Individual
- 20. A Simplified Approach for Evaluation Network/ Member Coalition Outcomes Capacity Capacity Image credit: http://www.cugelman.com/research/united-nations-web-network.htm
- 21. Member Capacity
- 22. Member Capacity • What do you want to get out of the network/coalition? • How do you justify your involvement? • How much time can you devote? • What can you offer? Individual Capacity Network Coalition Boundary spanners Skill/knowledge to work collaboratively Time/space to engage in network Time and Commitment to working in coalition Relevant skills and interest Relevant decision-making authority Desire to be part of something Strategic use of coalitions to fill critical bigger gaps and leverage resources toward achieving your mission
- 24. The Critical Organizational Capacities for Advocacy Initiatives: The Logic Model “Inputs” Initiative Logic Model Leadership: Management: Technical: Issue analysis/ research Improvement in the Quality of Living for the Community Media advocacy/ Public awareness raising Grass roots organizing Coalition building/ networking Policy analysis/ research Adaptability: Legal action Lobbying and direct policy- maker influence Access Administrative/ Regulatory window of implementation opportunity/ influence threat: •Offense •Defense Org. Culture (C) TCC Group 2008
- 25. A Few Key IOR Capacities Value Goal Proposition Destination (Why is this IOR the (What is the IOR right approach?) trying to achieve?) • Rules, procedures and decision-making • Action/Exchange-oriented • Leadership Entropy Bureaucracy (disorder) (rigidity)
- 26. Multiple Levels of Outcome Community Community Impact/ Outcomes Development of IOR Coalition/ Network Organization Benefits to Organizations Individual Benefits to Individuals
- 27. Capturing ‘Externalities’ • Sustained networks/relationships (Relationships hurt) • Reduced sense of isolation (Personality conflicts) • Better implementation/coordination of actual programs addressing the issue (More entrenched programming) • Cross-fertilization and Innovative ideas (Group think) • Other specific project goals met
- 28. Outcome questions for IORs in Emergency Settings • Has the IOR increased the ability to coordinate during varying stages of the emergency, including preparation? • Did the IOR reduce political complexity? • Did the IOR reduce or increase time and resource constraints? • Has the IOR increased the reliability of information available? • Has the IOR facilitated overcoming infrastructure challenges? • Has the IOR enhanced the well-being of personnel? • Has the IOR facilitated better relationships with affected populations and donors? • Has the IOR increased ability to engage with the media?
- 29. Conclusion
- 30. Contact Information Jared Raynor Director of Evaluation TCC Group www.tccgrp.com
- 31. Leveraging the Evaluator • Navigate power differentials • Use to: • Build (inform development) • Sustain (strengthen existing performance; identify inefficiencies) • Reflect (accountability for resources and prep for future) • Understand how to position yourself in a network • Help articulate goal destination and value proposition
- 32. Mobile Phone Principles of Good IOR Behavior 1 Getting the Right Plan • What do you need to accomplish? • Do you have appropriate bandwidth (value vs. congestion)? • Right amount of minutes/airtime?
- 33. Mobile Phone Principles of Good IOR Behavior 2 Aware of Surroundings • How should others contact you? You contact them? • Are you talking too loudly? Too quietly? • How much background noise is there? Photo source: whatisacellphonejammer.com
- 34. Mobile Phone Principles of Good IOR Behavior 3 • Strategic Use Do you have the right connections? • Are there new apps? • Are you playing too much Angry Birds? • Rate/assess your experience
- 35. Mobile Phone Principles of Good IOR Behavior 1 Getting the Right 2 Aware of Surroundings • How should others contact Plan you? You contact them? • What do you need to • Are you talking too loudly? accomplish? Too quietly? • Do you have • How much background appropriate bandwidth (value vs. 3 • Strategic Use Do you have the right noise is there? congestion)? connections? • Right amount of • Are there new apps? minutes/airtime? • Are you playing too much Angry Birds? • Rate/assess your experience
Editor's Notes
- #8: How we think about our impactSustainable marketplaceGrowing #s of participating orgsHelping those orgs be more effectiveUsing data to inform the sector as a whole
- #9: Michael Luca – Harvard Business School researcherhttp://www.hbs.edu/research/pdf/12-016.pdf+1 star = + 5%-9% RevenueOnly independent restaurants – no effect on chains.Chain restaurant market share declines with Yelp penetration.Yelp covers 70% of restaurants. Seattle Times covers 5% Michelin does not cover Seattle.Individual feedback influences the market.Smart store owners seek feedback and react to it.New markers of quality and reliability.More value created benefitting larger, more local group of restaurants.
- #10: This is what we’re trying to achieve.But how to we realize these Yelp-like effects within our marketplace?
- #12: Prompting question or image triggers a lived experienceStoryteller self-indexes story, gives meaning (can be done by others)Software detects visual patterns among storiesSoftware enables deeper dive into individual storiesPeople discuss patterns and story clustersAct on patterns, weak signals, phase shiftsStimulate beneficial patterns; dampen undesirable ones
- #13: Working! Library of Lived ExperiencesVisualizing Shifting Story PatternsDissemination to Official Aid SectorPossibility of Actionable Constituent VoiceUseful to GlobalGivingPromising!Rolling BaselinesUseful to DonorsUseful for OrganizationsUseful for Beneficiary GroupsInformation Challenges Existing FrameworksCross-Organizational ThinkingWorking on it!Stories facilitate decision-makingStories are basis for evidence-based policy changesFraming the ImpactPeer-to-Peer Knowledge Management
- #15: This is for an advocacy initiative. Implementing program work is not significantly different.
- #16: Concept of interorganiztaional relationships comes from Heather Creech and UniversaliaFrequently a conflation in the literature between coalition and networkCoalitions are almost always organized around member organizations, whereas networks are split between member networks and individual networks; however, in both cases, it is individuals that do the activitiesNatural gravitation toward increased structure or chaos leads to fluidity among the types of IOR that a single entity will go through.
- #17: Important to name these things.
- #18: So, up to this point you see a notion of convergence—by and large we are talking about nuance on a theme. Thus, the conclusion is one of nuance as opposed to distinct identity. To test whether this holds up, we must dive into the individual indicators and see if at the end they still lend us to this conclusion.
- #19: Not radically different questions for members to consider—they go through a similar decision-making process.
- #20: Community: It’s impact on; Increased visibility/knowledge of issue/public willBetter/less fragmented relationships with policy-makers and allies and reduced enemiesNew knowledge/development of good research (increased data) Overcoming important “sticking” points in moving an agenda/policyIncreased coordination of activities and better implementationDevelopment of IOR: Its position (power, centrality, capacity)Increased IOR capacity (e.g., clarity of vision; ability to manage/raise resources; visibility, etc.)Increased quality/prestige/ engagement of membershipIncreased collaboration between IOR members outside the IORMore rapid and organized ability to respondMore important to measure cohesion with network because there is a presupposed cohesion through formalization with coalitionsAbility to promote pair-wise relationshipsDensity as a measure (depth, strength and durability)Organization: Either direct or through individuals enhancing the organization based on experience in the networkNew relationship partnersIndividual: Personal development of skills, sense of belonging/identity, motivation, etc.Increased skills/ capacityIncreased legitimacy/ powerIncreased access to resources (while maintaining autonomy)Increased informationSense of belongingDecreased transaction costsUnit of analysis more likely to center on the individual/organization than the network as a whole.