Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

HTML

Joseph Haddad
Joseph Haddad
16,246 Points
Posted by Joseph Haddad
Joseph Haddad
16,246 Points

Should I disallow bots from crawling a user sign up/account creation page?

While writing my robots.txt file I assumed disallowing bots from crawling my sign up, log in, & forgot password pages was good practice. I assumed this because these pages are sacred to the users of my SaaS platform. Is my assumption blatantly wrong or does disallowing these pages not even matter?

1 Answer

Jeffrey James
Jeffrey James
2,636 Points
Jeffrey James
Jeffrey James
2,636 Points

99% of scrapers outside of the major search engines will completely ignore your robots.txt file.

Typically, user registration and auth systems are made secure by using 3rd party accounts (Google, Facebook, Twitter, Linked) to reg/auth, or buy confirming via email or using a captcha.

That said, if you're giving people some reason to sign-up and that may be a dishonest reason (getting coupons or something of value), people still figure out how to abuse it.

Joseph Haddad
Joseph Haddad
16,246 Points
Joseph Haddad
Joseph Haddad
16,246 Points

Thanks Jeffery,

That's what I figured as well.