MNT Only enable indexing of stable doc and dev developers doc (#22)

Author: lesteve

robots.txt

@@ -1,5 +1,4 @@
-# Do not let search engines index the PyData theme preview site
-# during the live testing phase.
-# https://github.com/scikit-learn/scikit-learn/pull/28353
 User-agent: *
-Disallow: /_pst_preview/
+Disallow: /
+Allow: /stable
+Allow: /dev/developers