Lokna’s laws of system operations

1. Respect the unknown. The fact that reality seems to defy logic or even the laws of physics only proves that there is at least one unknown factor. Expecting systems to behave as the manuals and your knowledge of the system dictates is only logical. Refusing to adapt to the fact that the system didn’t work as expected is not. If an unknown factor or factors caused an error once, it is likely to happen again under similar circumstances.

2. Respect what you don’t know. As the saying goes, the more you learn, the more you realize that you don’t know.

3. Only a fool deals in absolutes. Pity the fool, but don’t cater to his delusions.

4. Reading books, blogs and forums can only bring you so far. They are full of faults and misconceptions, and what is correct is often only correct from the viewpoint of the author. You have to adapt it to your situation.

5. Set up a personal lab and use it. Build it yourself, and build a lab reflecting what you use or want to learn.

6. You cannot succeed being on both the dev- and the ops-team. They may or may not share the same goal, but the path to reach the goal is rarely the same for both teams. Trying will only split your focus. Focus on what you do best, and let others focus on the rest.

7. Your lack of planning is not my emergency. Trying to make it so will only anger me.

8. Some people are otters, some people are rocks. Figure out which you are, live with it, or work on a way to change it.

9. Be responsible. YOU are at the top of your pyramid, and responsible for all of it, down to the foundations. Expect others to do their part, but be prepared in case they don’t.

10. Use independent testing software, and understand the test scripts. Your results will never be worth more than the test scripts behind the results.

11. There is no way to test production performance in QA or AT. No, there really is not. Until someone invents a way to create parallel universes for testing purposes, it will be impossible to accurately predict production performance.

12. Do not expect code to be delivered on schedule.

13. Do not expect servers and infrastructure to be delivered on schedule either.

14. Expect that the dev-team and customers will demand that you deploy on schedule anyway. Refer to 7.

15. If you are not on the DBA-team, do not waste time asking for admin access to the SQL Server. This will only anger the DBAs, and trust me, you do NOT want to deal with angry DBAs later on.

16. Do not try to use virtual machines to do the job of physical machines. A four core VM may perform well at low load, but don’t expect it to do the job of a 16 core physical machine at peak load. I know marketing will have us believe that this is possible, but it isn’t.

17. When you size servers, design for peak load and add 30% if you can afford it. If you cannot afford it, expect outages and abysmal performance at peak load. A 20% average CPU load does not mean that you can add 4 times your current amount of VMs.

18. Adding more VMs without adding more physical hardware will NOT increase performance, it will reduce it. Doing so will only increase the overhead on the already strained physical hardware. There are no exceptions to this law. If you still think there are, keep on thinking. Every time you add a VM, you have to recalculate your oversubscription rate and add hardware as necessary. Increasing CPU oversubscription above 200% is asking for trouble.

19. If you work for one of those companies that think you can just add ever increasing numbers of VMs to the same hardware hosts; RUN. Virtualization allows you to better utilize your hardware, but in spite of the price tag it is not magicware that creates resources out of thin air.

20. Do not believe the vm admin when he talks about max 5% hypervisor overhead. He forgot to add the 10% complexity overhead and the 10% general ops stupidity overhead.

21. Everybody lies, the question you should ask yourself is what do they lie about.

22. There will always be someone that is smarter than you. There will also be people that are a lot dumber. Sometimes the two are hard to tell apart.

23. If you delegate a task to someone else, don’t expect them to perform the task exactly like you would have done it. However, do expect them to deliver according to your instructions.

24. Some people like to argue for the sake of argument. They are usually very skilled at it. Trying to win an argument with such people is futile. That doesn’t mean you have to give in. There are other ways of winning than winning the argument.

25. If someone is yelling at you, they are usually part of the problem. This most certainly also applies to your boss, your boss’s boss and the president of the company. That being said, telling them that they are a part of the problem is not always a good idea.

26. Make sure to leave a paper trail. Covering your ass is a key survival technique, and there is nothing like a good subtle “I told you so” further down the line to brighten your day.

27. If you become the go to guy, people will be bothering you all the time.

28. It is OK to believe everybody else are idiots, but smart people keep such opinions to themselves.

29. Don’t play the blame game. Learn to tell the difference between finding the error and finding out who is responsible. As long as there isn’t any evidence of malcontent or criminal behavior, focus on preventing the error from happening again. Give people the opportunity to learn from their mistakes.

30. Shit happens. People make mistakes. Hardware will fail. Software will fail. Be prepared when it does.

31. Data loss is inevitable. Plan for that as well. Do not refuse to plan for data loss because you have the best backup strategy in the world, or because you think you RAID is infallible.

32. Backups alone are useless. Restores are all that matters.

33. Do not make excuses. Make changes instead.

34. There is no such thing as a fool-proof system. As soon as someone invents a supposedly fool-proof system, someone else will invent a better fool. Do not underestimate the collective stupidity of a large group of people.

Author: DizzyBadger

SQL Server DBA, Cluster expert, Principal Analyst

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.