Tuesday, July 31, 2007

Zimbra Crossing

A perfect mail server setup is a holy grail for systems departments in any organisation, whatever maybe it's size.
Whether to host the mail server locally, or outsource the hosting ? Will the data be safe ? Will the service be secure ? Will my provider be stable and is 1Mbps enough for my mails and internet access ? What solution do we adopt ? What will be the ROI/TCO/TLA ?
These and a zillion other questions arise in the minds of the systems, finance and other top management guys. In our company, we've been advising, deploying and maintaining a number of email servers for small organisations to large organisations with users ranging from only a couple to hundreds. Initially, we deployed custom home grown solutions using Postfix, Courier, Squirrelmail, OpenLDAP, MySQL, Spamassassin, ClamAV, Amavis along with a user management module, we wrote in PHP.

The installations were (are, in some cases) very robust, stable, scaling and had been extended extensively beyond the initial requirements (For eg, integrating with PureFTPd). Though we and our clients were very happy with these, we always felt the need for more features - to manage mail queues, better analysis and statistics amongst others.

In the meantime, there was a product which was making news as a featureful but open source email system - Zimbra. I took a look at the Zimbra, and was elated by the fact that Zimbra used the same open source components that we used to build our own system. Then, I was taken aback when I discovered most of the frontend was written in Java. Also, their complete IMAP/POP/Mailbox implementations are in Java - specific to Zimbra. Actually, I don't have anything against Java. In a couple of projects, I used to even enjoy the breaks I got while java code got compiled :-P. But nevertheless Zimbra wasn't for me. Yet!!

After a couple of months, when a customer asked for some very particular features. We discovered that it would take us aeons to implement that same features and integrate, given our limited resources. We finally bit the bullet and worked on deploying Zimbra for the client. Basing it on Debian GNU/Linux Sarge, we had a wonderful learning experience with Zimbra. We wrote several tools and utilities using which we migrated the users seamlessly and efficiently. Zimbra took off.

But, later we came across several bloopers with Zimbra - the distribution list didn't work, (btw, Zimbra's console management client works well), the number of new mails were misleading. Since the code was in java, we couldn't take the code, repair the same in a shorttime.

Recently, we upgraded Zimbra from 3.1.2 to a fairly large project Zimbra-4.5. It was such a wonderful experience - stage-by-stage application upgrade, it is very rarely used elsewhere.

... To be continued!!

Wednesday, July 18, 2007

Debian/Ubuntu, NSS, LDAP and Udev

We have a home brewn Single Sign-On implementation in our office which uses account information stored in OpenLDAP, exports home directories (NFS mounts) for clients using Automount. Our desktops run either Ubuntu Dapper or Debian Etch.

During the initial days when we deployed and provisioned the system, I faced a problem in which the desktops wouldn't boot.

udevd[1005]: nss_ldap: reconnecting to LDAP server (sleeping 1 seconds)...
udevd[1005]: nss_ldap: could not search LDAP server - Can't contact LDAP server
udevd[1005]: lookup_group: error resolving group 'nvram': Illegal seek

I originally thought since udevd starts up before networking in rc2.d, it isn't able to seek the LDAP server and hence is causing some problem. So, I changed the priority of networking from S40 to S02 and that appeared to solve the problem. But there were those udevd messages that still persisted.

Cut, six months later when Ubuntu Feisty Fawn was released and we installed the same. We still used to get the same problem. But no amount of changing the priorities helped this time. This made me take a hard look at the logs and it was then that I observed the last line in each set of logs.

udevd[1005]: lookup_group: error resolving group 'nvram': Illegal seek

Now, udev is set to create the device nvram at boot time and change group ownership of the device to nvram.
But we've setup the NSS service to lookup LDAP (in /etc/nsswitch.conf) for passwd, group and shadow. So, everytime udev wanted the group called nvram, a search for the group nvram was done in the local /etc/groups file and not finding it there, an LDAP seek was done (wow, PAM!!!) and either it couldn't contact the LDAP server (because network isn't brought up yet) or when contacted (as in our Dapper case) it couldn't find the group called nvram in LDAP.

Hence the solution would be to give udev what it seeks; The group "nvram"!

# addgroup --system nvram

Once that is done. A reboot confirmed this indeed was the solution!!! The moral of the story is that people creating udevd rules should take into account non-existant users/groups. And create them if not found. Also, a framework for the whole SSO solution is missing in the open source world, which is why Micro$oft is able to shove it's products to corporates. Let me know if any effort exists which does try to address the situation.

Saturday, July 7, 2007

Good Will Hunting!

Thanks to sachin, saw this movie a week or so back. I would rate this one as one of the best dramas ever I've experienced. The best thing about the movie are the dialogues. Sample this:



Though out of context, it might not make much sense. Watching the entire movie really gets one thinking. It got me! Reflecting!!

Wednesday, June 27, 2007

Why?



As usual, J.D. "Illiad" Frazer is right on top. But why ?

Sunday, June 24, 2007

Sampada (ಸಂಪದ) Love Day!

"Giving love to projects" is a concept innovated by the GNOME Project. The idea is simple. You identify an open source project which you're likely to give more attention to and work on it. You love your work, the work automatically gets done.

Today, I'd a similar opportunity to give love to. The project called Sampada (ಸಂಪದ) is a community portal run by a very dynamic and brilliant friend of mine. The portal is hosted on a server at a US Data Center and used to run Debian GNU/Linux Sarge. It had a very good uptime and it was being managed well. Quite a few services are hosted on the server including the website - PHP, Apache, the database - MySQL, Email for the domain (Postfix/Courier/Amavis/Spamassassin/ClamAV) and a management interface - SysCP, Mailing lists using GNU/Mailman. All these were working steadily without any major hitches from some time, but we felt we needed to upgrade the whole system to the newly released Etch distribution.

Typically this kind of entire system upgrade entails a huge amount of planning, dry runs, down time loss estimations. In addition to lots of caffeine to release the tension caused by all the nail biting edges the experience puts one through.

In this case, we decided that we need to upgrade the server and we'd devote some time this Sunday and that's that. No more planning than saying "we'll get the thing done!". Cocky! IMO. But we were only dealing with the world's most stable and most advanced (In many ways) platform and we'd enough experience to be so lethargic about not making those painstaking plans.

We started a bit late, but we started (unlike other days).

# apt-get update && aptitude dist-upgrade -y

The first thing, we did was to ensure a backup of the most important parts of the disk was taken and stored offsite. Took us about 2 hours. Then the initial run itself went pretty smooth and we've had most of the system replaced by newer versions. But there were a couple of packages, which had some issues. Turned out that the customisation that were done on the configurations of those packages - Amavisd-New, Proftpd weren't compatible with the newer versions.

But that was soon solved by slight brute force. What we did was to specifically seek the version we wanted to install on the system. This happened in the case of PHP 5, Apache2. Soon, it was done. And when we tested, voila it worked straight.

1 hour straight. Some questions asked. All tougher questions parried, prayed to god and hoped for the best. But the whole upgrade process was so Debianish!! Always reliable! Always works!!

Overall, we now have a spiffy and snazzy new operating system and an ecosystem of programs serving out very interesting, intellectual and colloquial thoughts to anybody who simply seeks from any corner of the world. Don't miss out!!

ಹೊಸ ಚಿಗುರು, ಹಳೆ ಬೇರು
ಕೂಡಿರಲು ಮರ ಸೊಗಸು

ಮಧ್ಯೆ ಚಿಗುರು ಗಿಡವಾಗಿ
ಗಿಡವು ಮರವಾಗುತಿರಲು,
ಆ ಗಿಡ ಮರಗಳಿಗೆ
ನೀರುಣಿಸಿ ಬೆಳೆಯುವುದಾ
ನೋಡುವುದಿನ್ನೆಂಥಾ ಕನಸು - ಮಂಕುತಿಮ್ಮ

Friday, June 22, 2007

Truly Plural!

I was working on a small shell script to generate stubs for a project we're working on. After arriving at a rough design and deciding on the class names, method names to be generated, I started generating the stubs (scaffolding?). And came across an amusing non technical issue. getBench() was fine, but getBenchs() wasn't. It should've been getBenches().

Uh, oh! There might be a tool that can generate plurals given a word ?

Doing a quick search threw up a perl library to achieve the same. Three cheers for Perl !!! I'm always amazed at the huge number of modules Perl has. Anyway's here's a simple wrapper I wrote for that library which you can use to get plurals for most nouns.

shashi@anacoluthon:~$ pluralize.pl bench
benches

While at it, I happened to notice there's an implementation for Python and also a Grammar ruleset for implementing elsewhere.

PS: Title Inspiration: "A truly rural frugal ruler's mural" - tongue twister

Tuesday, June 19, 2007

Of starting up blues

Though technically not a startup, we've been in a startup mode from quite sometime - say seven years. Oops! In seven years, history gets changed by companies younger than that. So, what does it take to build a startup and be successful while at it ?

Marc Andreessen has addressed some points in his blog, in which he points out the reasons not to build startups!! He hits the nail right on the head with each and every letter in that post. Ouch! That hurts!! Let me try to correlate.

Why do we start up in the first place ?
  • the opportunity to be in control of your own destiny
  • The opportunity to create something new
  • The opportunity to have an impact on the world
  • create your ideal culture and work with a dream team of people
  • money
The Opportunity to do. The Freedom to explore the Opportunities. The Opportunity to be Free. When compared to an engineering or an executive job, where you are one of the minions working to realise somebody else's ideas and opportunities, working for your own place under the sun is a very delicious idea. The freedom this portends, counters all the comforts that a cushy job can provide.

We've had numerous opportunities to explore over the years. Several ecsastic moments, several downfalls. The one big difference is that we didn't build to scale. We worked on the by now classical form of outsourced services. Having a replicable or assembly line model of products or services is going to make a big difference to the growth potential. We didn't do that!

Where can it go wrong ?
First, and most importantly, realize that a startup puts you on an emotional rollercoaster unlike anything you have ever experienced.
And what a rollercoaster!!! It ain't like nothing that can be experienced elsewhere. Further
You will flip rapidly from a day in which you are euphorically convinced you are going to own the world, to a day in which doom seems only weeks away and you feel completely ruined, and back again.
Know what??!! It need not be a day, even hours! minutes!! seconds!!! A positive side effect is going to be that you'll become a lot wiser, philosophical and equanimous. Of course, you need to have solid backing from your near and dear ones. And you need to have an outlet for your emotions without which you might end up in extremities. Like my friend, who killed himself a couple of weeks back :-(

In an established company -- no matter how poorly run or demoralized -- things happen. They just happen. People come in to work. Code gets written. User interfaces get designed. Servers get provisioned. Markets get analyzed. Pricing gets studied and determined. Sales calls get made. The wastebaskets get emptied. And so on.

Sigh! Those aren't something one person can do - day in and day out. Why didn't somebody tell me before ?

In a startup it is very easy for the code to not get written, for the user interfaces to not get designed... for people to not come into work... and for the wastebaskets to not get emptied.

You as the founder have to put all of these systems and routines and habits in place and get everyone actually rowing -- forget even about rowing in the right direction: just rowing at all is hard enough at the start.

And until you do, absolutely nothing happens.

Unless, of course, you do it yourself.

Have fun emptying those wastebaskets.

Thanks! Why am I pasting entire paragraphs here ? Because these are the exact thoughts I've been trying to put into words. Marc has articulated them so well. About the wastebaskets, "been there, done that!".
By that I mean that half or more of the people you hire aren't going to work out. They're going to be too lazy, too slow, easily rattled, political, bipolar, or psychotic.
Here, I've had quite a pleasant experience so far. Unless they are influenced by various factors, they do try hard, very hard to contribute. But, the issue remains that with all the paucity of resources, how is a startup going to manage it's people ? This according to me is the most complex challenge in the mix.
Fifth, God help you, at some point you're going to have to hire executives.

At what point do you hire executives ? And how do you compensate them ? Executives aren't cheap! They need to have a sense of belonging, if they're going to put in the effort. How do you address their hierarchy or position in the management ? What are the established roles in startups ? How do you shed your ego in order to work with someone with a more or less equal ego ?

Tough questions. If you've the answers, half the battle is won. The other half starts now.

Sixth, the hours
...
And even if you can help your employees have proper work/life balance, as a founder you certainly won't.
Stress. Strain. Pressure. And to think of it, I didn't think much of them even in my Engineering Mechanics classes. Maybe I'd flunked. Oh! wait :-p

It takes time for the culture of any company to become "set" -- for the team of people who have come together for the first time to decide collectively what they're all about, what they value -- and how they look at challenge and adversity.

In the best case, you get an amazing dynamic of people really pulling together, supporting one another, and working their collective tails off in pursuit of a dream.

In the worst case, you end up with widespread, self-reinforcing bitterness, disillusionment, cynicism, bad morale, contempt for management, and depression.

And you as the founder have much less influence over this than you'll think you do.

And by the time, a team settles. It's time for a new team to move in. Due to unlimited opportunities and these pressures, people move on. The camaraderie breaks. The work suffers. Customers get impatient :-)

Eighth, there are lots of X factors that can come along and whup you right upside the head, and there's absolutely nothing you can do about them.
Like our scheduled and unscheduled power cuts, lack of internet access, etc.

But, at the end of the day the thrill of coordinating all the activities, putting on various thinking hats, talking to people and getting things done is the bottom line for us - the founders/managers.