WebScraping.com

How to intercept secure network traffic from Android

2018-05-15T00:00:00+00:00

Often I need to reverse engineer how an Android app loads its data. If I can determine the relevant server endpoints then I can call them directly in future via a script and bypass the app. This process is more complex when the app uses HTTPS for network communication, which is becoming more common now that Let’s Encrypt provides SSL certificates for free.

To work with secure network traffic I use mitmproxy, which is an open source interactive HTTPS proxy written in Python. Installation steps are documented here. Once installed you need to determine the local IP address assigned by a router to your computer - this is different than the external IP address seen from the internet. On Mac I use the following to determine the local IP:

$ ipconfig getifaddr en0
192.168.0.105

Now start mitmproxy at this local IP and at a port of your choosing, such as 4444:

$ mitmproxy --listen-host=192.168.0.105 --listen-port=4444

Next step is to setup your Android device to communicate with mitmproxy:

Go to wifi settings and long press your connection
Select Modify network
Open advanced settings and change Proxy to Manual
Set Proxy hostname and Proxy port to the details used in the earlier mitmproxy command
Navigate the web browser to mitm.it and you should see a list of certificates to install - if not then the proxy settings are not working
Download the Android certificate

If the proxy is working then you should start to see requests from the web browser in the mitmproxy console window:

The network activity for Android apps is also viewable. If we open the Exchange Rates app then the following network activity will be triggered, which loads the exchange rate data used by the app:

This is a HTTPS request but we can still observe the internals because of the custom certificate installed.

Note that this approach will only work up to Android Marshmallow - in Android Nougat the security model was changed so that user installed certificates are no longer trusted by default. Also apps that use certificate pinning will not work, since this forces the app to only use a hardcoded trusted certificate. In these case more advanced techniques would need to be employed such as modifying the APK or tampering with the app at runtime with Frida.

How to automate Android apps with Python

2017-12-01T00:00:00+00:00

In a previous post I covered a way to monitor network activity in order to scrape the data from an Android application. Sometimes this appraoch will not work, for example if the data of interest is embedded within the app or perhaps the network traffic is encrypted. For these cases I use UIautomator, which is a Python wrapper to the Android testing framework.

To setup UIautomator you will need to install the Android SDK and set the ANDROID_HOME environment variable. On Mac I ran the following to install these dependencies:

$ brew cask install android-sdk android-platform-tools
$ export ANDROID_HOME="/usr/local/Caskroom/android-platform-tools/28.0.1/"
$ pip install uiautomator

To check whether your setup is working connect your Android device by USB and try the info command to check its status:

>>> from uiautomator import device as d
>>> d.info
{u'displayRotation': 0, u'displaySizeDpY': 640, u'displaySizeDpX': 360, u'screenOn': True, u'displayWidth': 1080, u'productName': u'hammerhead', u'currentPackageName': u'com.android.settings', u'sdkInt': 23, u'displayHeight': 1776, u'naturalOrientation': True}

If you get the exception EnvironmentError: Device not attached, then you need to enable USB debugging on your Android device in order to connect.

If the above command works then you are ready to start automating interactions with your Android device. Note that instead of a physical Android device you can use an emulator such as Genymotion, however I have found some popular apps will not run on emulators.

UIautomator provides a number of ways to select elements such as by text or class. Actions can then be performed on the selected elements such as click, drag, swipe, and keyboard input. One of the most useful commands is dump(), which returns the current XML layout of your Android device:

>>> d.dump()
<?xml version="1.0" ?>
<hierarchy rotation="0">
  <node bounds="[0,0][1080,1776]" checkable="false" checked="false" class="android.widget.FrameLayout" clickable="false" content-desc="" enabled="true" focusable="false" focused="false" index="0" long-clickable="false" package="com.google.android.googlequicksearchbox" password="false" resource-id="" scrollable="false" selected="false" text="">
      <node bounds="[0,0][1080,1776]" checkable="false" checked="false" class="android.widget.LinearLayout" clickable="false" content-desc="" enabled="true" focusable="false" focused="false" index="0" long-clickable="false" package="com.google.android.googlequicksearchbox" password="false" resource-id="" scrollable="false" selected="false" text="">
            <node bounds="[0,0][1080,1776]" checkable="false" checked="false" class="android.widget.FrameLayout" clickable="false" content-desc="" enabled="true" focusable="false" focused="false" index="0" long-clickable="false" package="com.google.android.googlequicksearchbox" password="false" resource-id="android:id/content" scrollable="false" selected="false" text="">
                    <node bounds="[0,0][1080,1776]" checkable="false" checked="false" class="android.widget.FrameLayout" clickable="false" content-desc="" enabled="true" focusable="false" focused="false" index="0" long-clickable="false" package="com.google.android.googlequicksearchbox" password="false" resource-id="com.google.android.googlequicksearchbox:id/launcher" scrollable="false" selected="false" text="">
                    ...
  </node>
</hierarchy>

This is equivalent to viewing the source of a web page and shows the attributes used by each element, which can be used by selectors to identify the correct element.

Here is an example script with UIautomator:

from uiautomator import Device
d = Device('014E05DE0F02000E')
d.press.home()
d.press('Maps')
d(text='Search here').set_text('NYC')
d.press.enter()

The script will:

connect to a device with the given serieal number (the shortcut from uiautomator import device should not be used if multiple devices are connected)
navigate to the home screen
open an app labelled Maps (in my case Google Maps)
type NYC in the search bar
and press the ENTER key to show results

UIautomator also supports many other features such as event listeners and relative selectors, which are well documented on the github page.

Web scraping with python book published

2016-02-01T00:00:00+00:00

For the last 9 months or so I have been working intermittently on a book covering the web scraping skills I picked up over the years for my work. It is now available on Amazon or directly from the publisher. Or can be found on BitTorrent.

Coincidentally another book on web scraping with python was released at the same time by O'Reilly, available here.

Heading to Oxford

2015-09-01T00:00:00+00:00

This year I will be fulfilling a lifelong dream of studying at Oxford University. For the next 11 months I will be completing an MSc in Computer Science, and during this time will work on projects for existing clients but probably need to limit taking on new clients.

How to scrape Android apps

2015-07-13T00:00:00+00:00

Google recently released the Arc Welder extension for Chrome, which allows an Android app to be run on the desktop. The aim of Arc Welder is to help make testing Android apps easier, but conveniently it also makes scraping Android apps easier too.

To run an app in Arc Welder we need to first download the app’s APK file. Google Play does not make this straightforward so I would suggest using an alternative such as androiddrawer.com. I often use the Exchange Rates app when travelling and was curious where they source their currency data. The APK is available for download here, and then when loaded in Arc Welder looks like this:

(Note that if you get an error saying WebGL is not supported then you need to force Chrome to support WebGL by enabling the Override software rendering list flag: chrome://flags/#ignore-gpu-blacklist)

Now that the Exchange Rates app is running on my computer I used Wireshark to track network traffic, which showed the following request was made:

This means that this app simply uses the Yahoo Finance API for their currency data, available at: http://finance.yahoo.com/webservice/v1/symbols/allcurrencies/quote?format=json. So to scrape the data from the Exchange rate app merely requires calling this API.
(If you want to learn more about how to use Wireshark check out their extensive documentation.)

I chose a non-controversial app for this blog post that uses a public API, but this same technique can be applied to any Android app that loads its content from a backend server, which ought to be most apps with data of interest.

Luminati

2015-06-24T00:00:00+00:00

These days I am often contacted by businesses asking if I want to try a free trial of their service. A recent one was Luminati, which claimed to have access to millions of IP addresses. They weren’t willing to divulge much over email and their website had less information than it does now, so we set up a Skype call. My contact was a salesman so he wasn’t able to answer technical questions, but gave me a good overview of what they are trying to do. Apparently they are an Israeli startup that built a peer to peer network called Hola, where users install a plugin to access content that is blocked in their region by downloading via other peers in the network. Now that they had millions of users they wanted to monetize this network by reselling it as a proxy service. Great idea, though when I signed up for a test account with Hola this was not clear, so I doubt most users are aware their bandwidth is being resold.

Unfortunately I found Lumanati’s costs are prohibitive for my typical usage:

A medium scale website requires roughly 50GB of downloading, which would work out at $1000 if downloaded via Luminati, so using this service would only be practical for small websites that quickly block IP’s.

Phone calls

2015-06-15T00:00:00+00:00

I have noticed that if for some reason a new client can not describe what they are after in an email, but want a quick phone call to clarify, this conversation will rarely develop into an actual project. I guess they want to pick my brain and then implement it internally or hire someone cheaper, so to limit the amount of time wasted I have started insisting an overview of the project be sent before setting up a phone call.

Bitcoin

2015-06-01T00:00:00+00:00

A few years ago I opened a US bank account so that US clients who wanted to pay by bank transfer could avoid needing to make an international transaction. This worked well until last month when clients started reporting their transfers were being rejected. I rang the bank (Chase) and after being transferred between a few departments was told I needed to come into a US branch with my passport to discuss the problem, which couldn’t be handled over the phone. Quite inconvenient because I don’t live in the US and didn’t plan to visit in the near future.

I expect the problem is receiving transactions from multiple client bank accounts raised some automated red flag, but won’t know for sure until next time visit the US. So now my US bank account was frozen without warning or explanation - as you can imagine I was not a happy camper.

This experience made me more sympathetic to the Bitcoin Libertarian philosophy where there is no centralized authority to get in the way of doing business. Consequently, I researched how the protocol works and added support for Bitcoin to my data store and invoice system. I originally budgeted a week of time to figure this all out but found the protocol much simpler than expected and in the end took just a weekend, certainly easier than an earlier integration I did with the PayPal API. The only complexity is that because transactions are anonymous I needed to generate a unique bitcoin address for each client so that I know who the transaction is from. This is how it works:

Find the current exchange rate between the clients currency and Bitcoin
Generate a bitcoin address to receive this transaction
When expected transaction is received at this address mark as paid

That’s it. Blockchain.info has a well documented API that can handle each of these steps. For exchange rates there is the Ticker API and for managing and monitoring addresses in steps 2 & 3 there is the Receive API.

Also then to display a QR code for the required transaction I used the Google Chart API:

Android database update

2015-05-15T00:00:00+00:00

A significant update to the Android Apps database is now ready, which now contains over 2 million apps (2,130,732 to be exact). If you have purchased this database previously you can login to your account to download the updated version for free.

Web Scraping or Web Scrapping?

2015-04-21T00:00:00+00:00

I searched my email and found over the last few years I received 76 messages from clients containing the text Web Scrapping rather than the usual spelling Web Scraping. And this is not unique to my clients - currently Google has 122,000 results for “Web Scrapping” compared to 447,000 results for “Web Scraping” - the correct spelling returns only 4x the number of results. So in light of this common spelling mistake I registered the domain webscrapping.com and redirected it here.