You’re working on a huge project.  Then the boss says, “Can we extend our system to running some other function on another website?”  You say, sure, why not?  Since we can just do a SVN checkout and tell Apache to use only one subdirectory, with a couple modifications we can have all of the features and functionality of our original system on this smaller subsystem.

Bad move.

Do you really need all of the functionality from your giant day-to-day system in your offshoot?  Does it need to share the SVN repository as your main project?  What happens when you need to make significant URL, IP, or system structure modifications?  Will the two branches work nicely with each other?

My suggestion: have your sub-system use APIs to talk to the big system.  Much cleaner.  The subsystem can live on its own, wherever it wants to on whatever hardware or architecture it needs to.  It can then make requests and pass data back and forth with your parent system in a similar way to how a client would.  This means that the upgrade paths of these two systems can diverge from each other and nothing’s going to break.  Need to introduce some fancy load balancing on one and not the other?  Go for it.  Need to move the child off to cloud-based hosting due to new business requirements?  No problem.

A little extra time spent separating distinct systems pays for itself in flexibility later.

August 20th, 2013

Posted In: Systems

Leave a Comment

This sounds like a pretty terrible topic, doesn’t it?  Stress Testing.  Sounds like it might be an interviewing method for new engineers.

Stress Testing is actually a great way to see if your servers can handle sudden jumps in traffic or sustained volumes of traffic (think of something going viral).  I learned today that a small wordpress site I set up earlier could easily handle about 80 simultaneous active users before it’s response times started getting too high and it went into a downward spiral of responsiveness.  I also learned that I could increase the number of CPUs and memory allocated to the virtual machine and it could handle significantly more simultaneous users.  This tells me that if the company running the WordPress site wants to start up a viral campaign such that 10,000 people are going to be hitting the site within 10 minutes, I better make some significant investments in load balancing, additional hardware, and software necessary to make that happen.  If however they don’t foresee any huge increases in traffic, then we can leave everything where it is and keep our costs down.

Do be careful running stress tests, and if at all possible, run them on dedicated hardware that will not affect production machines.  Your stress test could very easily overwhelm the machine you’re testing and if you don’t have good control on aborting the test, you’ve basically taken yourself offline due to a self-induced overload attack.

August 20th, 2013

Posted In: Systems

Leave a Comment

Sometimes you need to delete X number of records out of mysql table where the table may contain millions of records and/or gigs of data.

You could just write some code like this:

DELETE FROM screen_sizes WHERE url LIKE '%system_monitor.php%';

and call it a day.

But what happens when your query has been running for 30 seconds, shows no signs of stopping, and users page-loads are getting hung up because your MySQL server is backed up? What happens when it’s 10 minutes later and everything is still waiting on that query?

(Side note: If your table is MyISAM, a delete query blocks the table from further inserts and updates until the query is done. If you table is InnoDB, the table is not blocked but it does slow down to the point where you might have a cascade of inserts that never finish until the delete is done anyways.)

So how do you avoid this situation? Delete the records individually. This will make each query run very quickly and will put minimal load on the server. Write a little script that can be run every minute or so for as long as is needed. I’m also going to introduce a few functions in this code that will be explained in later posts, but you should still understand the concepts as-is for now.

<?php

include '../php_config/common.inc.php';

$query = "SELECT id FROM screen_sizes
			WHERE url LIKE '%system_monitor.php%'
			LIMIT 100";
$ids = getMySQLValues($query);
if($ids) {
	foreach($ids AS $id) {
		$screenObj = new screen_size($id);
		$screenObj->delete();
	}
}

?>

Putting a limit on the query as shown above helps the results come back quickly and limits the number of records you’re dealing with at any given time.

If you have 10,000 records you want to delete and you’ve setup the above script to run once a minute via cron job, it’ll take 1 hour and 40 minutes to complete the deletion procedure. Is that a lot longer than just running a delete query? Sure. Does it keep your system happy and keep users from ever knowing anything was going on? Yep.

In this example, we’re working with a very small set of records in a simple table but the same idea applies and is more relevant when trying to delete millions of records from a complex table with billions of records and dozens of indexed fields. In those cases, there’s no other way to keep a system responsive and still perform the operations you want in real-time without taking the whole system down for “maintenance”.

August 12th, 2013

Posted In: Systems

Leave a Comment