Give Up Internet!
  • Home
  • News
  • Pictures
  • Animated Gifs
  • Videos
SUBSCRIBE
No Result
View All Result
  • Home
  • News
  • Pictures
  • Animated Gifs
  • Videos
No Result
View All Result
Give Up Internet!
No Result
View All Result

The Robots.txt File of WhiteHouse.gov | America Do Not Want to Remember

by Sergey Page
January 21, 2009
Reading Time: 1min read
0
Share on FacebookShare on Twitter

White House Here is the previous Robots.txt file of WhiteHouse.gov

Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /expectmore/search
Disallow: /expectmore/query.html
Disallow: /results/search
Disallow: /results/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/text
The Full Index

And Here is the News Robots.txt file of WhiteHouse

User-agent: *
Disallow: /includes/
https://www.whitehouse.gov/robots.txt

If you are an internet people but not a developer, you may not know robots.txt

The robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard complements Sitemaps, a robot inclusion standard for websites. via Wikipedia

Tags: newsrobots.txtwashington.gov

About GUI

Give Up Internet is weblog focused on bringing you the best of the internet since 2008.

Recent Stories

  • What do I need to know to create an nft collection?
  • What is an internet meme?
  • Home
  • News
  • Pictures
  • Animated Gifs
  • Videos

© 2008-2020 Give Up Internet, Inc.

No Result
View All Result
  • Home
  • News
  • Pictures
  • Animated Gifs
  • Videos