We’ve fixed the site issues for now. I thought the “recent comments” feed had been disabled. It had not, and that was causing the bad performance. The hosting people redirected any hits on that feed to Google. Maybe Red State or Instapundit would have been a better choice.
Here’s the issue in a nutshell: when WordPress has to do anything with comments, it must plow through the 3 million comments that we’ve had at this site in the years since it started.
What would be ideal would be to have some kind of WordPress archiving plugin that would put 2.9 million of those comments in an read-only “archived comments” table, which would be used when someone pulls up an old post. The other 100K or so comments would be stored in a read/write “active comments” table.
I’ve searched high and low for something like that, and haven’t been able to find it. Does anyone who’s done big WordPress sites have a clue where we could find something like that?
Just Some Fuckhead, Thought Leader
Sounds like a plan. Just be aware I had a really good comment about a week ago that I don’t want to lose. keywords: splatkins diet.
Mark
It’s in yo mama’s house
Baud
It’s not the quantity of the comments but their quality that’s got WordPress down.
Howard Beale IV
Have you done an analysis of how many of the archived threads are accessed?
geg6
I don’t envy you, mm. I know nothing helpful but you have my sympathies.
Villago Delenda Est
Makes too much sense. Obviously, it’s creeping socialism.
dpm (dread pirate mistermix)
@Howard Beale IV: No, that would be the step we would take to solve this rationally.
fka AWS
What would be ideal would be to outsource the comments to disqus or livedebate or livefyre or whatever. That’s what the big sites do. Dragging 3 million comments around in the database isn’t really any kind of solution.
Does that suck? yeah, it does, but I’m not sure there’s a better option.
Mullah DougJ
Could we at least use Big Data to make wordcloud of all the old comments?
Santa Fe
DPM: You should get in touch with commenter “Tommy”, who seems to be a WP developer and posted some excellent comments on John’s open thread last night. See:
https://balloon-juice.com/2013/11/08/is-there-anybody-out-there/#comment-4706229
He went on to diagnose the problem pretty accurately, and has offered assistance to BJ.
srv
@Mullah DougJ: Yes, we need a big wordcloud of cudlipisms.
Just Some Fuckhead, Thought Leader
@Mullah DougJ:
OBAMA SQUEEEE! firebagger
mental illness cat
? Martin
Small bit of justice.
Texas.
Amir Khalid
@srv:
I’m okay with that, as long as we leave “maftoon” out of the wordcloud.
lonesomerobot
Not exactly what you’re asking for, but I hope you’re using this:
http://wordpress.org/plugins/w3-total-cache/
I use it on all the WP sites I develop. From what I’ve seen, it does improve performance.
fka AWS
In all seriousness, this is a good problem to have (he says as he adds to the problem).
dpm (dread pirate mistermix)
@fka AWS:
That would suck suck suckity suck, but maybe we’ll have to do it. Atrios uses Disqus. Anyone have thoughts?
dpm (dread pirate mistermix)
@lonesomerobot: We’re using it and we also have APC enabled. Thanks.
? Martin
Oh, and happy Kristallnacht! 75 years ago today, Germany got serious about voter fraud and those lazy Jews that were dependent on government, who believed that, that they were victims, who believed that government had the responsibility to care for them. Who believed that they were entitled to health care, to food, to housing.
Just Some Fuckhead, Thought Leader
@dpm (dread pirate mistermix): Disqus could be the worst commenting vehicle out there, if not for all the other ones.
greennotGreen
@? Martin: No, justice would be for Anderson to be sentenced to 25 years in prison. In what way is that different than his having kidnapped Morton and held him in his basement for 25 years?
Villago Delenda Est
@Amir Khalid:
I did a google search on “maftoon” and all I got were names, and I went in for several pages to see if there was any entry that might indicate it was some sort of slur. Prominent among the names was an Afghan folk singer who apparently has a substantial following.
So I’m not sure what Ms. Cudlip was trying to get across, except her own definition of the word which isn’t widely shared.
ranchandsyrup
Field trip to Russia to ask snowden.
? Martin
@greennotGreen: Don’t disagree. Why I said it was a ‘small bit’.
Amir Khalid
@Villago Delenda Est:
To this day I have no idea where that name comes from, or what m_c meant by calling me that.
Villago Delenda Est
@dpm (dread pirate mistermix):
I hate Disqus with the intensity of 1000 burning, angry suns. The main reason I stopped commenting at Eschaton is Disqus.
I prefer WP, even with its many shortcomings, to Disqus.
Baud
@Villago Delenda Est:
Ditto
Higgs Boson's Mate (Crystal Set)
Anyone care to hazard a guess as to how long old comments should be accessible? I’d say two weeks or a month.
Villago Delenda Est
@Higgs Boson’s Mate (Crystal Set):
Well, accessible, indefinitely. Allowing new comments? A month, at most. The active, writable database should be kept small, and the old comments archived away from the active threads. There should be a way to do this in a fairly elegant manner…other sites using other engines have ‘closed for further comments” features that allow you to still view the old comments, but not add to them.
Villago Delenda Est
@Amir Khalid:
She was calling you an Afghan folk singer?
She seemed to operate in her own little independent world very often, and this is a good example of that tendency.
Ruckus
@Villago Delenda Est:
Agree. Even given the current issues commenting here is about as good as it gets. The structure, not necessarily the content.
DPM The logic behind separating the comments into 2 databases, old and new, shouldn’t be all that difficult but I’ll bet no one has done it as a commercial product. Yet. It would slow down searches for comments if not restricted by say, a date range, but how often do people search for old or really old comments?
MikeJ
I’ll take a look at it. I’ll shoot you a mail if I figure anything out.
Higgs Boson's Mate (Crystal Set)
@Villago Delenda Est:
Thank you. I’m not strong with DBs or Word Press so I didn’t know. A “closed for further comments” feature would seem to be the best of both worlds.
fka AWS
@dpm (dread pirate mistermix): I am no fan of disqus, but just saying, comments seem to be not WP’s thing now. They’ve outsourced it. Look at wonkette, for instance. Do I like it? no, but if it makes the site work, it’s the way to go.
Elly
Some ideas here: http://wp.smashingmagazine.com/2012/09/12/secrets-high-traffic-wordpress-blogs/
Frankensteinbeck
@Amir Khalid (POTUS):
Oh, you missed it? She spelled it out once. She believes it means ‘a Muslim who no other Muslim would consider a Muslim’. She seemed to think it was in common parlance and you would know the word. I thought it was one of the most offensive things she ever said in a career of shockingly offensive racist bile.
lonesomerobot
@dpm (dread pirate mistermix): Benchmarking I’ve seen actually shows WP is faster than just about all other comment systems, including Disqus. There are plug-ins out there to close comments on threads, like this one (handy because you can close comments from a list of posts, and not having to go to each post individually):
http://wordpress.org/plugins/one-click-close-comments/
Amir Khalid
In the few years I’ve been commenting here, I’ve noticed that a thread rarely remains “live” for as long as two days. So I’d suggest automatically closing comments 60 hours (for example) after the front-pager has posted, and shifting the thread to an archive database.
pseudonymous in nc
@fka AWS:
God, no. The entire comments sections of blogs can get wiped out when the backend changes or gets “retired” or deprecated or something breaks. Steve Gilliard’s comments? Gone. Digby’s old comments? Inaccessible.
If you have a blog that values commenters and preserving their comments, then you don’t outsource. Atrios can use Disqus because the comments at Eschaton are basically a chatroom and have been for years, but for most sites, Disqus and all that stuff is for sites that want comments for pageviews but don’t care about commenters.
I’m not sure if there’s an off-the-peg plugin to archive old comments to a separate table, but sharding plugins like HyperDB might point the way.
@Ruckus:
For that kind of search, you’re better off outsourcing to Google’s spiders.
GHayduke (formerly lojasmo)
@Frankensteinbeck:
She was so damn obtuse I never figured out she was racist. Of course, I skipped over most of her posts because I couldn’t understand her word salad.
fka AWS
@lonesomerobot:
Benchmarking including what? sites with 1-2 comments per post, yeah it works. But I’d like to see a link backing up that comment. BTW, WP owns a certain comment system, so it’s obvious this isn’t their main thing.
fka AWS
@GHayduke (formerly lojasmo): the less we mention MC the better.
Santa Fe
As a web development director, let me agree with those observing that WordPress doesn’t excel at comment management and isn’t built as a forum tool first and foremost. Some open source CMS’s have much stronger native forum capability: Drupal is probably the best example. But as long as the commitment is going to be to WordPress as the CMS, probably worthwhile to look at implementing WordPress best practices for high traffic sites, which seems to mean third party commenting functionality as the long term solution.
EDIT: Please, just no Disqus! ;)
AdamK
It will be necessary to go directly to the WordPress Home Office and make use of those brass knuckles in your dresser drawer.
Frankensteinbeck
EDIT – Comment removed, @fka AWS is right, the less said the better.
cckids
@? Martin:
For a more uplifting anniversary, Happy Fall of the Berlin Wall!
I still remember turning on the tv that day & being just amazed at the sight of all those people partying on top of & around the Wall. One of the few “the world just changed” moments in time I’ve witnessed that wasn’t a tragedy.
max
Disqus has issues but it is worlds ahead of Livefyre. (Disqus works without all the *extra* JS crud that Livefyre has.)
At any rate, flat comments are nice, and they actually ought to be easier to handle than Disqus requests, but obviously WP has got the issues.
max
[‘Unfogged uses Movable Type and Ben has problems but never seems to have these kind of connection overload problems. On the other hand, the site load levels probably never reach BJ levels either.’]
Yastreblyansky
@dpm (dread pirate mistermix): I switched to Disqus at my (Blogger) place and have no complaints (I think it increased my comments from zero to statistical zero, maybe even a little more). I love using it at other sites, especially a busy place like Atrios (or here) where your little subthread can get so deeply lost, and having all of them accessible in a single place. Plus you can keep editing them forever.
Sir Laffs-a-lot
@dpm ask the folks at Wonkblog (if any are readers of this site); they use WordPress and I’d bet WaPo IT has a handle on THEIR comments threads.
Elizabelle
No Disqus and no threaded comments, please.
I like the clean, kind of plain interface here.
And please don’t lose any of the old comments. We’d lose General Stuck, for one thing. Twice.
Elizabelle
@Sir Laffs-a-lot:
The Washington Post has a HORRIBLE comments system. It used to be way better, as was the NY Times’.
With the Times, you used to be able to scroll through page 1, 2, 3, …16 … and with the NYTimes you can sort for “recommended” so you’re not reading Yosemite Sam.
Now it’s hit reload again and again. And slow.
Plus, never mind the trolls on WaPost. Pitiful. Mean. Dumb, but never in doubt.
Do not follow the WaPost’s lead on anything, please.
Harold
I’ve heard some good things about Jetpack http://jetpack.me/support/comments/ but no personal experience. Do you know if your db is InnoDb or MyISAM?
My experience with Disqus is a couple years old–but it would frequently overload our database when it got into a race condition when syncing comments. My former NPR colleagues seem to be happy with it these days though.
Have you considered Facebook comments? cuts down on trolls
cleek
Disqus would mean the end of the pie filter.
once upon a time, it was possible for GreaseMonkey to mess with Disqus comments – i had a pie filter for Kevin Drum’s site. but the latest iteration of Disqus puts all the comments in an external IFRAME, which makes it impossible (for security reasons) for GM scripts to access them. so, no more pie over there.
$0.02
I'mNotSureWhoIWantToBeYet
@Harold: Um, some of us still aren’t on Facebook (or Twitter) and have no interest in joining.
FWIW.
Cheers,
Scott.
Sister Rail Gun of Warm Humanitarianism
Is there some reason why you want to maintain the comments as a datatable?
I am way out of practice with the inner guts of WordPress, so there may be something in the newer versions that would make this unworkable, but why not take caching the pages one step further?
Make the older pages flat files.
It should be possible to write code to loop through the old posts and output a straight HTML file that you could then wrap inside the sidebars.
If the comments are a part of the archived page, they are no longer needed in the comments table.
Keith G
@cleek:
Sounds like a plan then.
I frequent sites that use Disqus and have never found it to be irritating. Maybe I just am not a pro commenter. What are the process issues that lead some folks to not like it?
Liberty60
No offense to anyone, but are there any comments here that really need to be saved for longer than a few weeks?
Or did the NSA get to Cole and force him to turn over our comments for review of any uspicious-Say ontent-Cay?
ryan
Take a look at the this comment/discussion software
http://www.discourse.org
Gex
@Liberty60: well, I’d want at least a year’s worth as we have the comment of the year nominations coming up.
Bob's Had Enough
I mod on an active Disqus-using site and, sorry, but this site is terribly slow in comparison. And Livefyre is problematic. We tried it for a while and dumped it.
I’d love threaded discussion. Discussions on this site are disjointed. Better to let keep similar content in one place.
What value would it give to go back 2, 30, 180 days and comment on a thread? Who would ever see ones entry? Archive threads after a few days. If someone wants to revisit an issue they need to bring it to an open thread before people will see it.
pseudonymous in nc
@max:
MT typically renders pages statically, even ones with comments.
@Liberty60:
Yes.
pseudonymous in nc
@Bob’s Had Enough:
Threading is bullshit. It was bullshit at Slashdot, it is bullshit at DKos, it remains bullshit in every implementation. It works in a few situations — for instance, focused QA forums that inherit threading from newsgroups — but it is a destroyer of any kind of rolling conversational environment online.
I'mNotSureWhoIWantToBeYet
@pseudonymous in nc: Bah.
Some people (like JC, who has said B-J “will never be threaded” at least once, IIRC) hate threaded discussion, so it’s a moot point, but …
You mentioned newsgroups. As I’m sure you know, threading was solved decades ago on Usenet. It worked. It worked really well for discussions that go on for several days, or weeks, or months. Or for very active discussions. For some reason, that knowledge was lost when everything was reinvented for the web. :-/
The problem with non-threaded discussions is they have to end and start over after a couple of hundred or so comments, or less, because of topic drift and because otherwise it becomes ridiculously difficult to see who is replying to who. If most everyone is just throwing up entertaining snark in a virtual pub, it doesn’t matter if there’s no threading. If “rolling discussion” is all that matters – sure, kill topics every couple of hours and miss the contributions of those who weren’t around when the topic was busy.
Slashdot had lots of problems – threading wasn’t one of them.
IMHO, of course. ;-)
Cheers,
Scott.
pseudonymous in nc
@I’mNotSureWhoIWantToBeYet:
What I meant by “rolling” is that with non-threaded formatting you don’t have to pick which sub-thread is the right place for a reply if you have a couple of directed responses, or split comments up into multiple replies so that they’re filed in the right place.
Blogs with multiple posts per day are always going to have some drift. As long as there’s a general consensus to stay on topic at the outset, that’s not a problem.
crack
@ryan: I second.