LISTSERV mailing list manager LISTSERV 16.0

Help for CHRONAM-USERS Archives


CHRONAM-USERS Archives

CHRONAM-USERS Archives


CHRONAM-USERS@LISTSERV.LOC.GOV


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

CHRONAM-USERS Home

CHRONAM-USERS Home

CHRONAM-USERS  July 2013

CHRONAM-USERS July 2013

Subject:

Re: Slow Page Load Times and Production Settings

From:

Michael Beccaria <[log in to unmask]>

Reply-To:

Data, API, website, and code of the Chronicling America website" <[log in to unmask]>

Date:

Mon, 8 Jul 2013 16:07:14 +0000

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (73 lines)

You folks have been very helpful. I will try the django-debug-toolbar app and the other suggestions shortly.

As for images, we are rendering png files. The load times seemed similar to standard jpgs and much faster than tifs and png's had a higher quality level for bi-tonal images which is what we are rendering up. We don't have the fast jpeg2000 decoder libraries. I personally haven't experiences very slow load times with images as of yet. I don't think it's as fast as LOC but I thought that might be because of your jpeg2000 library you are using to render the images.

I have 4GB ram allocated to this server VM and currently (using top command) it says I have 2.2GB used and 1.7GB free with 4GB of swap of which ~2GB is being used. I allocated 1.5GB of ram to Jetty via the JAVA_OPTIONS config settings. This used to be lower but solr choked when loading a large batch at one point so I bumped it up. 

Mysql has the defaults:
key_buffer              = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
query_cache_limit       = 1M
query_cache_size        = 16M

I added some lines to the my.cnf after investigating that these values were WAY low for innodb tables.
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 2048M

It's still slow but I'm starting to think it's a mysql index/memory/configuration performance issue. Vmstat 5 shows occasional swap usage in the si/so columns when trying to view the slow page. Innotop shows a really long query with a state of "Copying to tmp". The query doesn't wrap the txt so I can't see the whole thing. I don't have time now, but I'll explore more on the particular queries that are happening to generate this swapping.

I don't know of any other relevant settings that would indicate RAM usage. Is there any?

Any other Mysql advice?

Mike Beccaria
Systems Librarian
Head of Digital Initiative
Paul Smith's College
518.327.6376
[log in to unmask]
Become a friend of Paul Smith's Library on Facebook today!


-----Original Message-----
From: Data, API, website, and code of the Chronicling America website [mailto:[log in to unmask]] On Behalf Of Summers, Ed
Sent: Monday, July 08, 2013 9:42 AM
To: [log in to unmask]
Subject: Re: Slow Page Load Times and Production Settings

Hi Michael, 

In addition to the good advice you got from David and Chris, you might want to try the Django Debug Toolbar:

    https://github.com/django-debug-toolbar/django-debug-toolbar

Once enabled you get handy information about database queries, template rendering, etc as you browse through your web app. It is a generally useful tool for any Django project, if you happen to have others.

One thing I noticed is that images take a long time to render as well. This can prove to be a problem even when you put Varnish in front of Chronam, since researchers often interact with a long tail of content. Are you using the TIFF files? Also, how much memory does your server make available to Chronam?

//Ed

-----Original Message-----
From: Data, API, website, and code of the Chronicling America website [mailto:[log in to unmask]] On Behalf Of Michael Beccaria
Sent: Tuesday, July 02, 2013 6:50 PM
To: [log in to unmask]
Subject: Slow Page Load Times and Production Settings

I'm using the new code base and have a collection of newspapers that total over 200,000 pages. I know the LOC uses caching software (varnish) to speed up page loads and we currently don't have that running. On our site, the load time for the newspaper list page is probably close to 5 minutes (http://nyshistoricnewspapers.org/newspapers/) but the other pages load relatively quickly.

Is there something wrong in the code or is that expected given the queries that django is trying to execute and I need to install a cache to make it work faster?

This is part of a bigger question. What general recommendations do any of you have for putting this software into production with multi-million page collections? This could be server/network specific (Ram, virtualization, multi-server, etc.) or software specific (caching, settings, etc.). We want to release this site to the public relatively soon and want to gear it up to be ready to get hit by users.

Thanks so much for your suggestions and guidance.

Mike Beccaria
Systems Librarian
Head of Digital Initiative
Paul Smith's College
518.327.6376
[log in to unmask]
Become a friend of Paul Smith's Library on Facebook today!

Top of Message | Previous Page | Permalink

Advanced Options


Options

Log In

Log In

Get Password

Get Password


Search Archives

Search Archives


Subscribe or Unsubscribe

Subscribe or Unsubscribe


Archives

June 2019
April 2019
March 2019
February 2019
October 2018
August 2018
July 2018
June 2018
May 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
May 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
March 2015
January 2015
November 2014
October 2014
September 2014
June 2014
May 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
February 2013

ATOM RSS1 RSS2



LISTSERV.LOC.GOV

CataList Email List Search Powered by the LISTSERV Email List Manager