9 TED Talks That Anyone Working in synthetic data Should Watch
https://raindrop.io/dentunxlhq/bookmarks-43623738
This will allow Goggle (consumer-agent name GoogleBot) to go to just about every web site and directory, though at the same time banning Request Jeeves in the web site completely