Sitemaps, robots.txt and Structured Data: The Three Things That Get You Crawled, Indexed and Shown
Three small pieces control how Google crawls your site, what it keeps out, and how well your pages are understood: the sitemap, the robots file and structured data. Here is what each does, the common mistakes, and how to set them up right.

Three small, often-ignored pieces decide how smoothly Google works through your site: the sitemap tells it what exists, robots.txt tells it where not to go, and structured data tells it what each page means. None of them is glamorous, and all three are where a lot of sites quietly lose traffic — a blocked folder here, a missing sitemap there, a self-conflicting robots file.
This guide explains each one in plain terms, with the mistakes to avoid, written for business owners and teams in Egypt, Saudi Arabia and Kuwait. It builds on technical SEO from development to launch and feeds directly into getting every page indexed.
Key takeaways
- The sitemap is a complete list of your public URLs — submit it in Search Console.
- robots.txt controls crawling, not indexing; the two are often confused.
- Blocking a page in robots.txt can leave it in Google with no description — use noindex to remove, not robots.txt.
- Never block your CSS and JavaScript — Google needs them to render the page.
- Structured data (JSON-LD) labels your content so Google understands it and can show richer results.
- These files should update themselves as you add pages, not be edited by hand.
What does a sitemap do?
A sitemap is an XML file listing every page you want Google to know about, so it has a full map instead of relying on finding links. A large site uses a sitemap index that points to smaller sitemaps — one for pages, one for products, one per language — which keeps each file clean. The limits are generous: up to 50,000 URLs and 50 MB per file. The rules that matter: include only real, indexable URLs (no redirects, no 404s, no blocked pages), include image references where useful, keep it updated automatically as pages change, and submit it once in Search Console. A sitemap does not force indexing — it makes discovery complete and reliable.
What does robots.txt control?
robots.txt is a small file at the root of the site that tells crawlers which paths they may or may not crawl. Use it to keep crawlers out of areas that waste their time — internal search, faceted filters, admin paths — so they spend their crawl budget on pages that matter. Two rules save a lot of pain: never block your CSS and JavaScript (Google needs them to see the finished page), and point to your sitemap from robots.txt. If you want your content to appear in AI assistants, keep robots.txt open to their crawlers too; we build this into every site.
The robots.txt versus noindex trap
This is the mistake that costs the most. robots.txt controls crawling; noindex controls indexing. They are not interchangeable. If you want a page out of Google, you must let Google crawl it and see a noindex instruction — or return an X-Robots-Tag noindex header. If you instead block the page in robots.txt, Google cannot read the noindex, so the URL can still appear in results with no description ("indexed, though blocked"). Rule of thumb: robots.txt to save crawl budget on pages you do not care about; noindex to actually remove a page from search.
What is structured data and what does it earn you?
Structured data is extra code (usually JSON-LD) that labels your content in a language Google reads directly: this is an organisation with these contacts, this is an article with this author and date, this is a product with this price, these are the breadcrumbs. It does not change what visitors see; it removes Google's guesswork and can unlock richer listings. Common, useful types:
| Type | Use it on | Benefit |
|---|---|---|
| Organization | The whole site | Name, logo and contacts in Google's knowledge of your business |
| Article | Blog posts | Eligible for article treatment and clearer understanding |
| Product / Offer | Store product pages | Price, availability and reviews can show in results |
| BreadcrumbList | All deep pages | A clean path shown instead of a raw URL |
| FAQPage | Pages with Q&A | Read by Google and AI assistants (rich FAQ results are now limited to a few site types) |
Structured data is also how your pages become quotable by AI assistants — part of being found beyond the ten blue links.
These files should maintain themselves
The common failure is a sitemap or structured data edited by hand once and never again: it drifts out of date, lists dead URLs, and loses Google's trust. On a well-built site the sitemap is generated live from the real pages, structured data is produced from the page's own data, and robots.txt is version-controlled — so they are always correct as the site grows.
How Nilex helps
Every Nilex site ships with a live sitemap index that updates itself as pages are added, a correct robots.txt that protects crawl budget and keeps private areas out while allowing search and AI crawlers, and structured data (Organization, Article, Product, Breadcrumb and FAQ) generated from each page's real content — plus an llms.txt so AI assistants can read a clean summary of the site. It is part of our SEO-ready website service. For how all of this turns into indexed pages, see getting every page indexed.
Frequently asked questions
Do I need a sitemap if my site is small?
Yes. It is quick to add, makes discovery reliable, and lets Search Console report on your pages properly. There is no downside.
Will robots.txt remove a page from Google?
No — that is the common trap. robots.txt stops crawling; to remove a page use noindex (and let Google crawl it to see it). Blocking in robots.txt can leave the URL in results with no description.
Is structured data a ranking factor?
Not directly, but it helps Google understand your pages and can earn richer listings, which lifts clicks. It also makes your content easier for AI assistants to quote.
Should I block AI crawlers?
Only if you do not want to appear in AI answers. For most businesses, being quotable by assistants is an advantage, so we leave those crawlers allowed by default.
Technical SEO From Development to Launch: Building a Site Google Can Actually Find11 November 2026 · 6 min
Get Every Page Indexed: A Clean Google Search Console With No Coverage Errors11 November 2026 · 6 min
Server and Deployment SEO: Redirects, Status Codes, HTTPS and Canonicals11 November 2026 · 5 min
