| Summary: | bugzilla robots.txt blocking web crawlers such as archive.org | ||
|---|---|---|---|
| Product: | Libre-SOC Website | Reporter: | Jacob Lifshay <programmerjake> |
| Component: | website | Assignee: | Luke Kenneth Casson Leighton <lkcl> |
| Status: | CONFIRMED --- | ||
| Severity: | normal | CC: | libre-soc-bugs |
| Priority: | --- | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | All | ||
| NLnet milestone: | --- | total budget (EUR) for completion of task and all subtasks: | 0 |
| budget (EUR) for this task, excluding subtasks' budget: | 0 | parent task for budget allocation: | |
| child tasks for budget allocation: | The table of payments (in EUR) for this task; TOML format: | ||
|
Description
Jacob Lifshay
2019-12-22 01:24:13 GMT
yep this apparently is quite common, web-crawling of bugzilla can be pretty heavy so mozilla set up a default that banned pretty much everything. i'm not so bothered so have set it to "Allow /" (In reply to Jacob Lifshay from comment #0) > a good template to use > would be Mozilla's bugzilla robots.txt: > https://bugzilla.mozilla.org/robots.txt just copied it entirely, just... because :) (In reply to Luke Kenneth Casson Leighton from comment #2) > (In reply to Jacob Lifshay from comment #0) > > a good template to use > > would be Mozilla's bugzilla robots.txt: > > https://bugzilla.mozilla.org/robots.txt > > just copied it entirely, just... because :) Thanks, sounds good to me! if you have more time, it would be nice to also fix #149 |