busy
incredibillfeed, I noticed that he was one of the panelists on the Search Engine Strategies forum on bot obedience. Turns out he is the creator of CrawlWall, which is a firewall for crawlers (aka bots, the programs that pull web content so it can be indexed by search engines). He and I have debated the merits of certain types of blocking policies on WebmasterWorld. For example, he favors blocking server farms, claiming they are homes for malicious bots that scrape site content (putting it on other sites that claim it is their own), not to mention click fraudsters. My response was that while this is certainly possible, it's best to ban access to one's site based on observed bad behavior, rather than presumed identity. After all, the entity accessing the site may be perfectly legit, just using a user agent no one has ever heard of before (because it's new). Also, IP addresses change hands, so what could have been a server farm last week could be a subscriber pool this week.Anyway, based on what he writes I can tell he is very passionate about what he does. However, I couldn't see myself trying to create a company to do something like that. The reason is because the problem he is trying to solve is not intrinsically interesting to me. This is not to say it isn't important or doesn't have value, but it's not something I'd care to spend most of my time on. And when one is trying to get one's business off the ground, especially when it entails a good deal of software development, one must be willing to spend most of one's time on it. There is just too much competition, especially from companies with very deep pockets. If they sense there is a market to be served, they'll easily jump in and steal your market share. (You may get lucky and they'll buy you, but you still have to put in enough effort to create a business that someone would be willing to buy.) Also, in this type of business, there is a lot that could go wrong, despite the best of intentions. There are a whole lot of potential pitfalls due to unexpected corner cases, unreliable ISP service, etc.
In case you're wondering why I wouldn't want to do something like create a crawler firewall, but spend a fair amount of time repeatedly playing the same measures from a piano piece, or repeatedly practicing a dance pattern, it's because even though those activities may seem boring I derive value from the end results. However, as I wrote some time ago, I'm not passionate about either of those – not to the extent that I would spend most of my time on any one of them. I try to spend equal time on several activities, or if that's not possible, enough time to meet the goals I've set for each. But with regards to dancing, I don't dance much; perhaps once a week outside of my lesson, and I'm starting to think that after the showcase I won't sign up for any more lessons for a few months. It seems as if dancing is a luxury I can't really afford right now, financially or timewise. When I'm more actively involved in dancing it feeds on itself and I want to be even more involved, but it just seems like it will go on the back burner in the immediate future.
