Keep It Up

shadeland 2003-04-15 18:30:24 ET So I'm going over what it takes to host SK or something with simliar functionality. It will not run on shared hosting. The database calls alone are intense, and will monopolise a server. A site like this, even with limited images, probably pushes a couple hundred gigs of bandwidth a month. A site like this requires a dedicated server, which I have. It would need MySQL 4.0.x with "select caching", which caches often hit selects, which would really save on database calls. I know there are several efforts do something identical to mine, but I've got unique resources to pull it off. Anyone know what the status is?

CyberrPunk 2003-04-15 18:32:43 ET the status is 'hopeful', from what i can see. lots of ideas, but this is gonna need organization.

shadeland 2003-04-15 18:42:36 ET Is there any centralized discussion? I went off to enjoy the fine day, and lost track.

Alice 2003-04-15 18:44:37 ET My biggest fear is that the whole will break up into many smaller groups. 5 ppl go there, 30 go here.... never to have the same mix again. I'll try to keep track of who is doing what. I'll join the one I like best. Like you said there is a lot of work to do.

shadeland 2003-04-15 18:47:40 ET I'm thinking that a number of factors will really only leave one around. Talk is cheap, after all. I think several attempts might be made, but the obsticles will prove to be insurmountable.

Alice 2003-04-15 18:53:39 ET That is why I feel a transfer of ownership would be best. Not many ppl are going to be willing to make everything from scratch. Hell even Moxie didn't build it from scratch. In reality someone owns the site but no one owns the community, with out the ppl there is nothing but an empty domain name. I think moving the community is the only way anything can really succeed. Destroying it only to rebuild isn’t the answer.

shadeland 2003-04-15 19:05:18 ET From scratch isn't even the hardest part, it's actually the funnest part. It's the resources required to host such a site that I think will be the hardest part. Like I said, it won't run on a shared hosting site. It requires it's own dedicated server, and a fast one at that. Otherwise, performance would be at the whim of the other people sitting on that shared box, and the database calls would hammer the system. Getting a dedicated server ain't easy. It typically requires a year-long contract and it's a couple hundred a month, at least. Probably a bit more with the bandwidth costs associated with a site of this type. The hardest part about that is it usually requires a year-long commitement/contract, which not a lot of people are willing to sign. There may be sites that don't require that, however. I've built sites like that before and run them; it's what I do for a living. I even wrote an O'Reilly book on scaling web servers with a network device. I could put something together, but I don't know if I could have all the functionality completed by May 15th, hence why I need help with the coding. It's just the time involved.

m0xie 2003-04-15 19:06:14 ET Hell even Moxie didn't build it from scratch. Uhh, actually, I did. :) The page layout styles are similar to iam.bme, but the code is 100% from scratch.

shadeland 2003-04-15 19:10:25 ET BME is one some funky compiled code Shannon wrote. I'm not sure what he uses, I don't believe it's C though. I think I saw a post about visual basic, but I'd be suprised if that were it. SK runs on PHP/MySQL on FreeBSD, am I correct?

Alice 2003-04-15 19:18:47 ET My mistake, I was under the impression you were given the initial framework. I don’t like coding, it is hard for me to imagine what went into an undertaking of this magnitude. Since I know, that you’re watching I just want to say. Thank you.

Meursault 2003-04-15 19:18:59 ET he used Powerbasic

m0xie 2003-04-15 19:19:08 ET BME was written in some strange version of compiled BASIC. The fallacy is that because it's compiled, and uses custom database structures as opposed to a relational database backend, it must be faster/more efficient. You are correct that SK.net is written in php using a MySQL backend. It can run on any platform that supports MySQL & PHP. The site is currently hosted on a dedicated Linux (shudder) server. Up until sometime in Februrary, it was hosted on a dedicated FreeBSD server; however, the Linux server is much, much cheaper.

shadeland 2003-04-15 19:41:10 ET 10 print "internal database" 20 goto 10

Alice 2003-04-15 20:25:57 ET I’m learning P-Basic so I can program microcontrollers for custom lighting applications. 24 pins on a chip, and that is all I have to worry about. Other than simple “if-then” statements I can’t imagine doing anything with Basic. A friend of mine was trying to get me into PHP but I didn’t have a use for it on my site, since all that I wanted could be done with CSS. If I had known it would come in handy I would have picked it up. Now all I can offer is limited monetary assistance.

slaad 2003-04-17 13:33:32 ET How's PHP for speed? As compared to say... mod_perl?

shadeland 2003-04-17 13:50:58 ET Hard to say. I've never run into any situation where PHP was the bottleneck; it's typically MySQL (or another RDBMS) or something fancy PHP does, such as instense GD calls for image manipulation. So I believe it's realtively fast, although die-hard perl fans will probably say mod_perl is quicker.

slaad 2003-04-17 14:10:25 ET Yeah, I didn't want to start a language war or anything. I just have this preconcieved idea that PHP would be slower than mod_perl as mod_perl is pre-interpreted.

shadeland 2003-04-17 14:14:13 ET I imagine that's true in situations where the language is doing a lot of the heavy lifting, such as intense math (image manipulation) or using internal database sctructures (sorting and indexing), but most of the heavy lifting in a site like this is done by the database back end.

m0xie 2003-04-18 05:40:16 ET A strong, flexible, abstract architecture is what makes for a good dynamic web site; the backend implementation almost isn't important so long as the architecture can support scalable load levels. sk.net is broken up into very small modules that do only as much as they need to. Even though php is an interpreted language, the execution time required for a particular module is virtually unnoticable. A highly tuned backend database ensures that access is also not a bottleneck. Most queries complete in milliseconds, even on tables that are quickly approaching 1 million rows. As I said earlier, abstraction is key. If php were to become a bottleneck, it would be trivial for me to rewrite speed sensitive modules in another language such as python or even a compiled executable. This could be extended even further to use something like FastCGI, where requests would be handled by prespawned processes. Additionally, there is a fair amount of caching that could be added to the page display routines that would significantly reduce database calls. As you can see, there are many, many ways that sk.net can scale. I believe this scalability component is missing in "other" online communities. ;) PHP and MySQL are often looked down apon because, like virtually all tools, there's a wrong way and a right way to use them. It's very simple to write a simple php script with a simple database backend and have it function without any problems under light to moderate load. However, it takes more experience and know-how in order to build something that can realistically scale. A table scan on 10 rows will complete significantly faster than a table scan on 1,000,000 rows. ;) (And yes, I realize I'm preaching to the choir here..)

shadeland 2003-04-18 11:01:29 ET Before I knew much about PHP and MySQL, I helped administer a popular punk message board (I had the hosting and did the sys admin work). We got around 600 kilobits of sustained traffic during the day (that's quite a bit). The problem was the guy who wrote the code was used to ASP/Microsoft SQL, and he designed the tables so that the posts andthe parent threads where in the same table. To generate the main page, it had to do a full select on the entire table (close to a million rows), which hammered the system. We routinely had to export the older rows to keep the site's performance from going south. The solution, of course, is to have the parent posts and child threads on separate tables, so select's would only be done on smaller tables, and the larger table would only be accessed when a particular thread was brought up.

Return to shadeland's page