The site looks really nice!
My guess about what is happening: your MySQL index size exceeds the amount memory you have available to MySQL. This could be either because of a configuration in your my.cnf, or it could be a physical limitation of the machine (e.g. there is too little RAM physically on the machine to fit the index).
If you can tweak your MySQL configuration to add RAM, that should help, but if you can't, I think putting a reverse proxy / cache in front of it should alleviate most of the problem. We have some similarly slow pages on one of our instances of the software that load in less than a second from the cache.
How many newspapers do you have listed on that page? Other options would be to either remove some of the query-heavy fields from the table or to paginate the list, depending upon how much you want to dig into the templates.
From: Data, API, website, and code of the Chronicling America website [mailto:[log in to unmask]] On Behalf Of Michael Beccaria
Sent: Tuesday, July 02, 2013 6:50 PM
To: [log in to unmask]
Subject: Slow Page Load Times and Production Settings
I'm using the new code base and have a collection of newspapers that total over 200,000 pages. I know the LOC uses caching software (varnish) to speed up page loads and we currently don't have that running. On our site, the load time for the newspaper list page is probably close to 5 minutes (http://nyshistoricnewspapers.org/newspapers/) but the other pages load relatively quickly.
Is there something wrong in the code or is that expected given the queries that django is trying to execute and I need to install a cache to make it work faster?
This is part of a bigger question. What general recommendations do any of you have for putting this software into production with multi-million page collections? This could be server/network specific (Ram, virtualization, multi-server, etc.) or software specific (caching, settings, etc.). We want to release this site to the public relatively soon and want to gear it up to be ready to get hit by users.
Thanks so much for your suggestions and guidance.
Head of Digital Initiative
Paul Smith's College
[log in to unmask]
Become a friend of Paul Smith's Library on Facebook today!