Comments Problems Resolved … I Hope

CQ readers know that we have had nothing but frustration with the comment process here, a problem that has long predated the recent site upgrade. Over the last couple of weeks, we have tried everything we could think to resolve it, and I’ve spent more time on this problem than I have on my actual blogging. One reason for that is that the same problem commenters have in posting on CQ happens for me when loading new posts as well. The server load has driven performance to a crawl several times a day, and none of us could really understand why. The Blog Studio and Hosting Matters have put a lot of effort into trying to make the comments work properly again.
Eventually, the good people at Six Apart (creators of Movable Type) explained a number of facts about the commenting process that made the problem clear. We knew that the comments script invoked a rebuild script, but what we didn’t know was that it invoked a rebuild for all archives as well as the index files. Earlier, I had applied a fix that addressed a different rebuilding process, but it didn’t apply to the archives.
Six Apart suggested that I try a plug-in, Rebuild Queue. This defers all automatic rebuilding queries for archives until a scheduled program runs. I’ve set it for once per day, overnight, when not many comments get left on the blog. That way, all rebuilds will happen at once, allowing the site to operate much more quickly and the rebuilds to occur more efficiently.
Hopefully, this will be the final fix. Let me know if you see any improvement, and thanks for all your patience!
UPDATE: Actually, I’ve had to try another kind of fix, also from Movable Type. RebuildQueue works too well — it also stops rebuilding the RSS feeds and the main index file that creates the blog, which means that I’d have to rebuild both manually whenever I create a new post. I’ll try this other fix for a while; it seems to be only slightly slower than the plug-in.
UPDATE II: If you got an error message with the word “abaltach”, it’s fixed now. I wrote the wrong script name into the program, and it took a while for me to find it. Sorry!

38 thoughts on “Comments Problems Resolved … I Hope”

  1. Ah, Captain. I know you went to Michelle Malkins’ designers to “upgrade.” And, while I don’t go to her site; I did see people who comment, here, warning you that her site was not “comments’ friendly,” anymore. To the point where some of your people said they don’t go there, anymore.
    Now, Glenn Reynolds IS one of the sites I also visit daily. And, his latest post is to comment that for AUGUST, which had been a “slow month” … his site just set records. It’s quite a record-setter if you see more visits, and more “page views” in a slow month. Of course, Reynolds “wondered why.” But I think the Net is picking up traffic, just in general. And, this will get “even better,” ahead.
    It also means that even when people are on vacation; using their computers is on par with driving. MORE happens during vacations. Not less.
    Anyway, the site with the best comments arrangment, in my book is LITTLE GREEN FOOTBALLS. So, I’m surprised you haven’t gone to Charles Johnson, and asked for help.
    Who knows? If traffic, here, were to go, insane; and you’d get hundreds of posters showing up; wouldn’t it make sense to have an idea of how Charles Johnson does it? His site is perfection, itself.
    And, yes. I’ve seen the posting problems. Even where I know why some people double-post. And, it’s part of the problem of the site.

  2. Carol,
    The problem isn’t in the CSS, which is where the designers work, but in Movable Type’s base program. Charles created his own system, so it’s not quite the same thing.
    Believe me, I know what a headache this has been. I’ve lived with it 24/7 for quite a while.

  3. Feh.
    I bet lots of people who post have given it up, tonight. Because the system got stuck, again.
    Even though Charles Johnson went his own way; his system is perfect. Yours is not. Whatever the “movable tripe” system is; it’s not as smart as Little Green Footballs.
    And, when you switched, you got people who read your site, and comment, coming on board telling ya that Michelle Malkin didn’t improve her site. Instead? She lost loyal customers.
    You mean Charles Johnson doesn’t sell his product? He should. It’s gangbusters.

  4. After watching you struggle with moveable type, some of the seams exposed (I’ve lost 5 or 6 posts to really weird error messages from your database backend) make me think the problems are with MySql or your php compiler, not moveable type. The stuff I’ve seen (aside from the misnamed resource) indicates loss of your database connection pool or inability to obtain a database handle (oversubscription problem?).
    Good luck working it out. This certainly is not a good advertisment for MT and I hope the solution is simple when you find it.

  5. Ed, Carol knows the difference between CSS and the back-end databases.
    She’s saying MT sucks compared to other platforms, and it does.
    Word Press, Johnson’s system, and God forbid, even Blogger’s is much more stable when it comes to handling large volumes of comments.
    Michelle Malkin’s site design was a definite step backwards.
    Hot Air’s great, MM.com not so much.
    The new Captain’s Quarters CSS, however, looks great. You got value for your money.
    I kinda miss the blue (my favourite colour), but this is still a great design, and excellent branding.
    Now how long will you keep on trying to drive a square peg through a round hole until you realize that one of the reasons MT isn’t king of the hill anymore is they’ve been eclipsed in database competence?
    If you could get Johnson’s system, if he parted with it, you would have the best commenting system out there.
    His is better than that used by Fortune 500 companies or the Blogging specialists like Google Blogger, Word Press, and MT.
    Loads FAST, uses AJAX (a type of Javascript, which works faster because doesn’t require a full page refresh to update information) to retrieve new comments, regularly handles posts of 1000-2000 comments with speed for user and blogger alike, and degrades gracefully for users without Javascript to the old still stable system.

  6. No, the problem is in the MT programming. They sent me an unpublished adjustment to the comment program to keep it from rebuilding all of my indexes. Once I corrected the script-name error, it dropped the comment process time from 2.5 minutes to 20 seconds, which isn’t great but isn’t bad either.
    Carol, after spending all evening working on these fixes, your “feh” and recitation of the obvious about the lack of perfection here is not really appreciated.

  7. Ed, Carol (who if you check the archives I disagree with almost inevitably) was trying to be helpful.
    Okay, you don’t appreciate it. So I’m being critical then.
    In accordance with your commenting policy, I’m allowed to disagree. It sure as heck is on topic.

  8. I don’t think that would be a mysql or php problem, as those are stable and obviously they are only responding to the requests sent. More likely the software, Movable Type, is not built soundly.
    I would highly suggest migrating to a different software. Even wordpress can handle comments better than Movable Type.
    It’s actually rather ridiculous that there are so many problems commenting. Your site is nothing but text!! It should run as smooth as silk, but everytime you try to post it takes a good minute to go through. As you mentioned in your post, any software that rebuilds the whole damn archive on each new comment has problems. That’s ridiculous.
    It would be cake to transfer to new software. As far as I can tell you probably have a small database (structure wise) as there shouldn’t be anything except time, date, poster, content, feeds, and comments. This site should run so fast!
    Truthfully captain, you don’t even need blogging software. All they do is package together a stylesheet and commenting engine, which you can do by yourself. Joomla runs better than this, and that’s not even supposed to be blogging software.
    Don’t be scared of changing software… The task seems large at first but in the end its worth it. I like your site, but commenting has been a problem ever since my first visit.

  9. Hey, that last comment took about 2 seconds to go through… maybe there is hope =)
    Now all you need is an ajax driven commenting system =)

  10. I like your site, but commenting has been a problem ever since my first visit.

    Ditto. Since the first visit.
    I came here because the content was great and the insight is quite balanced.
    I don’t mean middle of the road, I mean fair and emotionally mature.
    This is great. Your blog is excellent.
    The software? Almost EVERY other site I visit works better and often they’re just standard Word Press installs.
    Because you’re not using and MT template, you can go anywhere.
    I’m sure the people at Movable Type are very nice people. And if they can fix it, great.
    Just saying I’ve been here off and on for a year and a half and it’s never worked well yet.

  11. I’m pretty sure we have the problem licked. I’ve finished playing around with the fixes — it’s been a long time since I did even rudimentary programming — and it seems to be running properly now.
    It’s actually a very big deal to switch software, and the archives almost never make it. If I wanted to start from scratch with no archives, and if I wanted to pay for another redesign for a new software system, I’d do it … but neither sounds very pleasant at the moment. As it is, I’m probably going to post a reminder about my tip jar sometime this week.
    At the moment, though, it looks like the comment system is working properly, and other than that, I like MT. It’s got a great interface and many different ways to customize it. I use WordPress at Heading Right, and it’s nowhere near as good on the interface. Comment management is much more limited. I’ve used Blogger, too, although not for a couple of years — and it’s not even in the same ballpark.

  12. I visit two sites numerous times each day; “Lucianne.com and CQ”. If I have time, I visit others, but these are daily “Must Reads” for me. Glad you do what you do and do it so well. Thanks.

  13. In the past, it was only when the process completely stalled that I knew it worked and my post would appear, eventually, if I lived long enough. 🙂

  14. SoldiersMom —
    Thank you!
    Christoph —
    The problem was cumulative. MT wants to rebuild all of the archives and the indexes on every comment, and as the blog gets bigger, the rebuilds take longer. The problem gets worse the longer it goes, and as more comments get added at the same time, the scripts start to hang and run away on the server.
    The fix I put on tonight only rebuilds the post itself (which MT should have done from the beginning). That’s why it’s moving faster now — because it’s not rebuilding everything in sight, and because it’s not, the scripts won’t hang any more.
    At least … I hope so!

  15. Can I get a special signature chime whenever i post a comment?

    Something, oh … happy yet regal.

    Bad time to ask for stuff?

  16. OOC,
    I’m thinking of the Imperial Margarine fanfare, but I don’t know if you’re old enough to remember it. I am … unfortunately…
    Christoph,
    The color matches the new graphic. I was aiming more for sepia, but this is as close as it gets. I wanted something with less contrast than the navy.

  17. It looks good. I was wondering about the new graphic too. Does it have a story?
    Hey, have you considered a “stylesheet switcher” so people can choose blue or green, but with everything else the new style?
    It would be SO easy to do. Like trivial. I could do it.
    I have on my two websites.
    And you could switch from new to old graphic with the switch. One click of a button and everything would change back and forth, but with the databases completely untouched, just background color and the graphic.
    Not essential, but just a thought. If you did want to do it, it’s really quick.

  18. Like on The Jawa Report, but without the Star Wars and your second theme would be much closer to the first.
    Not all dark and hard to read. Virtually identical, but with the old graphic for your old fans and blue colours and that’s it.
    The main content part would be the same, sidebars would be same except for colours and the header images (i.e., “THE CROWS NEST” would have a blue paterned background).

  19. Yes, that will do perfectly, thankyou. I can’t wait.

    I am old enough to remember that one and also “it’s not nice to fool with mother nature” which i think was a Parkay commercial. But, arguably, the greatest commercial of them all (okay, beer commercials and A1 insurance notwithstanding) was “now, that’s a speecy-spicy meatball!” which was a great rolaids or tums commercial but I don’t remember the jingle for it or even if it even had one so I’ll settle for the margarine.

    Sorta sucks to be old enough to remember all these. Kleidsdales in the snow means it’s Christmas time.

  20. You know, it’s odd… I enjoy leaving posts if I have something worthwhile (in my eyes at least) to add. But in the end, I come to this site to read YOUR take on issues, Cap’n… not because I can leave mine. I belong to a forum or two, and if I get the urge to spread my Wisdom around I get it out of my system there.
    That being said, I had tons of problems with the old TypeKey system… never could get it to work on at least half of the blogs that used it. But for this post, your site is having no problems at all… at least the preview loads in about five seconds. (Three the second time!)
    As for Charles’ system: It too is a work in progress… he’s been tweaking it for weeks now. Mostly cosmetic, a bunch of improvements as well, but he also has had some significant failures to contend with.
    It will be a good system when he gets it working right over a longer period of time than a few days. In the meantime, I wouldn’t consider using it until it is more mature…
    No offense, Charles!
    Software that works on blogs with a couple hundred posts and a few hundred comments can get away with running such relatively small databases for quite a while without problems… but when you get a site as large as this one, Michelle’s or Charles’… then the weaknesses are going to show up in high relief. I assume it’s not just generating text, but spam filters, all kinds of stuff. Frankly, I’m surprised it works so well…

  21. I couldn’t get to this web-site for the past week with my Comcast connection. I kept getting a message basically saying the DNS couldn’t resolve the address. Tried a web based proxy and got the same message. Tried another proxy and it connected me to the site. I got up early today and found things back to normal.
    A question for the techies. Was this problem caused by Comcast or Captains Quarters/Hosting Matters?

  22. Capt.
    My last post only took about 3.5 seconds from “summit” to being excepted and page reload. That’s about 1/100 of what it usually took before. Keep your fingers crossed 😉

  23. RE: Chimpy (September 1, 2007 4:46 AM)
    …DNS couldn’t resolve the address and the relative success of different proxies.
    Since Comcast has been transitioning its services as it rolls out to different networks (which in this day and age is constant with large ISP’s), it was probably a regional DNS server that had an invalid, perhaps stale IP in its cache. Depending on the proxies you tried, they could be on the same trunk and relying on the same upper tier DNS server to do name translation. If Ed’s technical support was changing DNS records during the debugging process, then it’s entirely possible that his IP address could have been changed (then changed back) with a stale entry not being updated fast enough. I’m too lazy to do a DNS lookup history to see if that was done. 😉
    Oh, the fun of the innertubes. Without further info from Ed, we cannot know the culprit.

  24. Glad to hear you found ~something~ and made the change, Ed.
    Thanks for hanging in there until you found relief – for yourself and us!
    ~grin~

  25. LGF does have a great comments sys but Charles is a Techno Geek of the first order. This means his site has some great features but is also subject to crashing and burning on occasion.
    BTW, CQ is on my daily too. Don’t comment often but I read voraciously!

  26. god i had no idea the work the mice on the those little treadmills were doing to keep sites running.
    how i solved the posting problem a long time ago. i keep my favorites list open as i scroll down to where i want to go. click on CQ and occasionally leave a comment. the minute the first bar appears to show its transmitting i click on CQ on my favorites list again. I go read the rest of CQ. my comment always shows up instantly.

  27. I’ve had a couple of hiccups with Comcast here, too, but only on one of my laptops. I keep having to flush the DNS when the problem appears. And yes, it’s the Vista laptop. Hasn’t cropped up in a few days, though.
    I haven’t made any domain or server changes in a few months, though. I don’t think that problem was on our end.

  28. Patience is virtuous……like Patrick, I found that the simple “refresh” solved most of the waiting period.
    I was hoping that Ed had intalled this “comment delay” to frustrate the newbees that don’t take a moment to think about what they are writing before they hit the publish/post button.
    While I like the site upgrade Ed, it will always be the sagaciousness of your content I look forward to daily.
    As we used to say in the printing business—–press on.

  29. P.S. Don’t touch a thing Ed, it’s working to perfection (at the moment) Or, at least save these settings if you’re still tweaking.
    And thanks again.

  30. Your site is looking nice and easier for these tired old eyes to read.
    If it’s of any consolation I value your blog for the information and your opinion and not so much the comments Heh! 🙂

  31. Capt Ed,
    Just posted a comment over on one of the top placed threads and the posting time was pretty darn speedy! I think you may have the fix and if so, you better batten down the hatches, because I foresee Comment Posting totals of your threads reaching 150 and 200 before long.
    I LOVE this new setup, Cap’n Ed! (

  32. Hey, Ed, do you happen to have Bad Behavior running, particularly an older version? If so, might want to upgrade it. I noticed (on my tinier site) that the older version slowed down everything, but, particularly comments.

Comments are closed.