Certification Demand and Quality
A few years back I got three silly tech certifications: Certified Information Systems Security Professional (CISSP), Microsoft Certified Database Administrator (MCDBA), and Microsoft Certified Systems Engineer (MCSE). That sounds like an odd mix of certs for a programmer; I have them because I’m a mercenary an independent developer and I also do architecture work. Many programmers regard certifications with scorn, for good reason. But merit, or lack thereof, is one piece of the puzzle. For me the main questions around a certification are:
- Is there market demand for the certification?
- Does the certification truly verify knowledge or ability?
- Should I look for it when hiring?
Certification providers make a lot of claims about jobs and improved incomes due to certification. Most skip over the sticky issue of correlation versus causation. Do certifications boost your income, or is motivation behind both? Or, less graciously, is the data simply crap from people seeking your hard-earned cash? We can get an idea of demand from Dice.com:
Is this the MCDBA? Or is this the CCNA? Or is this the IRA? I thought it was the UK!
By looking at total cert numbers, there’s clearly some demand. Strangely Java certifications seem to be an exception. Please let me know if I made a mistake there. Absolute numbers tell part of the story, but a better test is comparing the number of jobs that mention a given certification to the total number of relevant jobs. The chart below is my attempt at that:
Dice.com - % of relevant jobs mentioning certification
The data is flawed since it’s impossible to query “all relevant jobs”, but I tried to make sure it was sane for each cert (I posted the actual searches as a comment). I think this reasonably captures market demand for certifications, including the fortunate reality that programmers can safely ignore them. Outside of programming demand is stronger; in security the CISSP is notably successful. There are two points worth mentioning. First, certifications still have an effect on customers and jobs that have not explicitly asked for it. Whether that matters depends on your career: people who are starting out or interview frequently for contracts stand to benefit more. Second, the more clueful a company is, the less stock they put in certifications. Some hardcore, academic, or start-up workplaces downright shun certifications, especially programming ones.
Which brings us to the next point: do these certs prove anything? For the ones I have obtained, all in multiple-choice format, I feel safe answering “hell no!” Certified people might truly know the subject or they might have studied for a month or they might be good at multiple-choice tests. Basic answering techniques go a long way toward passing the tests. The ol’ elimination, contradiction, and parsing answers out of the tests themselves gets you almost there. Add a few hours studying targeted prep books and materials and random people could obtain random multiple-choice certifications, provided they test well in general.
When I took the tests my programming work involved security, so I fulfilled the CISSP requirement for three years of experience. I had decent knowledge of certain areas covered by the Microsoft stuff and the CISSP, but in other areas I was utterly ignorant. If these tests were up to snuff, I would have had to work much harder. The CISSP, given its better reputation, was a let down - I found it looser than the Microsoft stuff, which is dodgy to begin with. As luck would have it, in the week prior to the CISSP test (i.e., the study week) I started reading Harry Potter, which is like crack in that it takes your mind off everything, only more addictive. In the evening before the test I thought “crap, I just lost $400 in test fees.” I wondered whether to even show up - “surely the mighty CISSP is harder than the Microsoft stuff and you can’t pass with no study, little sleep, and a head full of Hogwarts.” Not so. Hence, in a Groucho Marx sort of way, I lost all respect for multiple-choice certifications.
In fact, the CISSP unites two negative aspects of certification: a faulty testing mechanism coupled with a heavy-handed experience requirement. I oppose criteria like degrees and years of experience, which correlate weakly with talent and job performance. They also strike me as unfair in a certification - let each employer decide how to value such things independently of the knowledge ‘verified’ by the testing.
My conclusion is that certifications based solely on multiple-choice tests are at best misguided and at worst shams. I don’t see how they could be fixed either. There is a fundamental disconnect between filling in the right blank and getting technology work done. For programming certifications the multiple-choice format is ludicrous. Hence these silly tests will remain a marketing and money-making scheme fueled by naive employers, who put faith in vacuous credentials, and the professionals who play along for supposed career benefits. Or maybe they’re fueled by the naive customers of the employers.
The final question is whether to look for certifications when hiring. In my experience there’s actually some positive correlation between certifications and the quality of job candidates, but it is too weak to be useful. I can’t use certifications as a weeding tool, they don’t guarantee a phone interview is worthwhile, and I would never rely on them for a hiring decision. Multiple-choice certs are silently ignored in my hiring process. The reality might be different for more stringent, hands-on certifications like the Cisco Certified Internetwork Expert (CCIE).
As to whether you should get certified, it’s obviously too particular a decision, but I hope this information helps. One of the best pro-certification arguments is that if you study the stuff anyway, for fun or profit, then why not go ahead and take the tests? It helps you focus somewhat and you get the warm fuzzy I-passed feeling. Coupled with the potential advantages in the market, maybe that’s good enough reason to take them. For those in markets where certifications are strong, it may be necessary. But on principle I’d sooner avoid them and for most programmers I don’t think they make sense.
Certifications that actually proved some knowledge and talent would be useful for job seekers and employers. Where possible, we must shun broken certifications and pressure vendors into adopting a valid testing scheme. This necessarily would involve practical tests, like the ones in the Red Hat Certified Engineer test or the CCIE. Both are head and shoulders above the ones I took. For programmers, certification could use on-the-spot, time-capped programming assignments. The results could be a mixture of the source code produced (the most important piece) and the outcome of an automated test suite. That’d be pretty useful during the selection process: the ability to see source code produced in standard conditions of temperature and pressure, before you sink hours into a candidate.
What do you say?
July 28, 2008 at 1:11 am | Filed Under Programming
Subscribe to blog
Comments
15 Responses to “Certification Demand and Quality”
Leave a Reply
Alright, here are the Dice queries I used to measure the relevancy:
For CISSP, I measured “Information Security” versus “Information Security” AND CISSP. Similar searches return results in the same ball park.
For RHCE, I measured total jobs by looking for “Red Hat”.
For Cisco stuff, total jobs are the ones containing “Cisco”. That distorts the result for CCIE, but trying “Cisco Senior” was even worse.
For Java, I measured total jobs via “java programmer or java developer or java architect” since Java alone returns even more hits. This made the cert relevancy a little better, but it’s still negligible. I searched for acronyms and fully spelled certs, but it sounds like Java certs didn’t catch on. Good for them.
For the MCSE, I compared “Active Directory” AND MCSE versus “Active Directory”. There are some bizarre and crummy jobs that ask for the MCSE, and by mentioning AD I at least filtered the absolute slog out.
For MCDBA, “SQL Server” compared to MCDBA.
For the MS dev certifications, I measured total jobs via “Visual Basic” OR C# and measured cert jobs my asking for MCSD OR MCPD, etc.
I do agree with you! Not only in certifications but also in college degrees, there is an overall simplicity to it all. I had friends that passed the exams on some programming classes but don’t really know anything about programming. That’s why I am very disappointed with Education in general. If you want to learn something, learn by yourself and if you really have to work with what you are learning even better, you’ll learn it quicker
In my faculty I had 1 or 2 classes that had a project and so you had to use what you were learning and the last semester of the last year was extremely disappointing as it focused mainly in theoretical stuff and stuff from the 70 and 80. I had no final project, nothing, just pass the exams and voila I had my degree. Wasn’t easy, but it was because I had a lot of classes that were useless and those were the ones that complicated stuff
Nice post
I must tell you that certification personally helped me in knowing Java better when I took SCJP and scored a close to 100 score. But I won’t like to title my resume as Srikanth –the certified Java programmer. Moreover Steve Yegge will never look at my resume
I’m planning to learn a bit of networking and system administration and my best bet is to go through a course that a certification demands me. I will come to know what’s relevant and what’s not about something on which I have no idea otherwise. And I will also end up having a feeling that I know a little bit of networking if I pass the exam with a great score. If I just manage to pass it, then I’ve really not learned much. I aim scoring big ones when I take certifications. Imagine a MCSE guy taking SCJP and passing it with a score of 85+, I can assure you that he’ll be able to read Java code and understand (assuming it’s well written).
So personally if I want to learn about something new, certifications definitely help me not lose my focus. But I won’t expect it to give me monetary rewards or promotions in my job.
I have to agree with your conclusions. I’m a .NET developer and decided to take the SCJP to try and keep my skills up to date. I passed reasonably well but in retrospect it’s useless because no Java job actually requires that kind of knowledge or assumes it to be a minimum requirement. What’s more important is which frameworks you have knowledge of because that’s what really does the heavy lifting in Java programming these days.
Getting these certifications are not only enough. You should aim at getting a high score and moreover expertise in that subject. I agree, there are people that have certifications but are incompetent, but they are the ones that go for these exams with the sole intention of getting monetary. You should think well before enrolling a for certification and training program and see where your interests lie so that it does not go in vain.
Particularly, I have a SCJP certification I got when I started working and the knowledge I aquired studying for it has helped me on some very few and far between situtations on the job (but at least it wasn’t totally useless).
I had a voucher for a SCWCD certification I bought as soon as I completed SCJP, but I decided not to take it given that most of the market worked with technologies which made SCWCD obsolete, and I don’t regret it (even if I lost money).
I don’t know about the other certifications, but I’m all about doing it when you have to study the topics very deeply and they can make a difference on your expertise in work. I just don’t think Java’s certifications fit very well in this requirement.
The certifications are useful in proving to complete strangers to your field (and IT in general) that you have at least some basic skills and understanding of stuff you’re supposed to be working with. Anyway, the non-IT people can’t tell bluff by talking to you, the certifications are often the only gauge they really can read.
And it’s not just like ending up as a witness in court, where CISSP (and MSc) are easy way to establish at least some kind of credibility. In networking, Cisco’s lesser certifications are often required for gigs with non-Cisco gear, especially if the client is quite oblivious about IT, as you have to know at least the basics to get them.
That’s the important part. The point is to establish the lowest acceptable level. That’s why it can feel they’re easy to get, especially if you’ve long and broad work history. If you are good, your common sense starts to exceed the certifiers’ expectations.
On the other hand, some certificates are more in demand than others. For example Cisco used to require their resellers to have some minimum number of certified people. Anyway, I think CCIE is somewhat different, not only because getting it isn’t very easy, and because it has it’s own ecosystem going on, to the point where some employers carefully avoid mentioning it in their job ads unless they really need it, just because that tends to drive the salary expectations significantly up.
Thank you all for the thoughtful comments. I hope to reply tomorrow as it’s a bit late here now. cheers.
Nice post. Informative as always.
I’m considering taking the SCJD cert, specifically because it would be impressive (maybe) to take it before becoming a junior in high school.
Should I? It could be a waste of 300 dollars if it’s not going to be useful or impress anyone.
Just to throw in here: I think the reason the Java certs are so unpopular is they are generally not representative of skills in “real world” projects.
Having passed the Programmer and Developer tests, I can say with some confidence that there is little that applies to day to day problems beyond simply syntax questions. Particularly in the JEE world, whether a Java developer is a “fit” for a job has more to do with tech stacks than Java the language. If, for example, there were a Spring certification, I would expect it to carry more weight the SCJP — where a good bit of the exam is focused on Swing idioms that a lot of Java developers never use.
Specifically, I would be interested in seeing the BEA(BEA/Oracle) and IBM “Enterprise Application Developer” and derived certs tossed in here too. While it certainly would shift the focus as much as the MS ones do, I think it would be an interesting exercise.
These are good points. Here are some thoughts:
@AntonioCS: exactly, the classic disconnect between CS education and building software. Joel Spolsky had a nice post on this, I linked to it in the ‘lucky programmer’ comments.
I like that the Java stuff gives out scores. Both the CISSP and the MS stuff are pass/fail, so you don’t really know if the person scraped by or aced it. Scores == more information, and maybe if you see a 90-something % on the certs it might be a data point on a resume, as opposed to a blend “MCSD” which means nothing to me right now.
However, I’m not so sure somebody who passes the SCJP would jump in and understand a real-world Java app. The trouble is that the SCJP works at too low a level: syntax and base classes and snippets. Would the person recognize Visitor and Builder and Memento and all that? Hibernate, spring? Would they understand the role of cookies, how the JavaScript is calling into the server, and so on?
And at the end of the day, you also want them to _write_ good code, which the cert is not measuring
@Marcos: I think the certs can help ensure you cover some ground. Take .NET, I’ve been working with it since Beta 1 way back in 2000, I know it pretty well, and I’m sure I’d learn if I prepped well for the cert. Problem is, you don’t have to prep well, heh, so maybe I’d just procrastinate, take the test, and pass anyway since they’re not rigorous. heheh. Then you add the money and the time studying arcane stuff that you should just google or look up in documentation. All in all, I’m not crazy about the multiple-choice certs at all.
@Vanya: Agreed, but I think the certs are *failing* the people who must rely on them to gauge expertise. That’s part of my beef, the fact that the tests could be _improved_ a lot if companies were willing to put more work on it than just crank out these current tests. Like the Red Hat Certified Engineer - it’s not a CISSP, but at least it’s practical. I’d like to see more of that.
@Prad: Hmm, I hesitate to give advice, “first do no harm” and all
Here are some ideas though. Since you’re so young, the cert might help you establish credibility. Also, later on if you’re applying for jobs, it might be cool to have the cert with the date you obtained it on. That will make you look a bit more experienced on the resume.
But I guess it really depends on what you want to do. College? Grad School? Google? Small Company? Start up? hehe. Also, it depents on what $300 means to you. Big deal? Nothing at all? Might be fun just to take the darn thing.
Also, what else cold you do with your time? How about some open source? That might impress more and open more doors than the cert. I’m pretty sure it’ll teach far more too.
@Richard, @Robert: yea, I do agree that the real-world development and frameworks have rendered the SCJP sort of outdated / insufficient, and that does explain its small presence.
Yea, looking at the BEA/IBM stuff would be interesting… so many topics, so little time
Thank you all for the feedback again.
Terrify to certify…
Любопытный анализ востребованности различных профессиональных сертификатов. Хотя, в общем, ничего неожиданного…
Yup, cert would certainly cover some grounds, but more importantly, it helps you to go past the job recruiter to the REAL interview. How you do in the interview would probably depends much more on your skills, but certs help you to get there in the first place.
Because you can talk all day to the job recruiter about your technical stuff, but at the end of the day, he/she would only know the keywords in the job ad.
Unless of course, they do provide you with some screening test. But then again, that would be something like cert ain’t it (study for the test) ?
[...] difficult to collect data on whether certifications help IT pros land consulting gigs, but Gustavo Duarte recently analyzed job postings on Dice.com that mention specific vendor certification…. For most of the certifications he examined, less than one quarter of the relevant job postings [...]
Of course, the ne plus ultra certificate is when you have contributed patches to the Linux kernel or to open source projects.