Brown aug11 bsdmag
0 likes505 views
This document is the third part of a series on establishing and managing an open source certification program, focusing on the technological aspects necessary for its success. It outlines key topics such as website security, survey creation, collaboration tools, and the importance of effective communication among various stakeholders. Additionally, it describes the processes for exam construction, delivery, scoring, and the metrics for assessing the program's overall effectiveness.
Brown aug11 bsdmag
- 1. LET’S TALK What It Takes Starting and Running an Open Source Certification Program, Part III This is the third part in our series on what it takes to run an Open Source Certification Program. In Part I we discussed “People”, the kinds of people you will need to help you run a Certification Program for your most excellent software. T hen in Part II we talked about Processes – how to can alert you when you have a problem such as a denial get the Certification Program set up as a business of service attack or an unauthorized login. In some cases, and how to start up the exam development effort. your hosting provider can provide additional protection This month we’ll look at Technology – what kinds of tech by filtering upstream traffic. Keep in mind that any you can use to get things done. Most of the items in this configurable security measures should be under some article are about well known topics- websites, monitoring, form of change control. This will help you if you decide to collaboration tools, and so forth, so we won’t go into get your Certification Program accredited by ANSI/ISO as elaborate detail. If you would like more detail, drop us a line we discussed in Part II. at [email protected] and we’ll be happy to oblige. Surveys Website You’ll occasionally need to poll your user community It seems every business has a website these days. It’s for information, such as what kinds of certifications to the easiest, simplest way to hang out your shingle for develop, or what the name(s) should be. Taking a survey everyone to see. Because it’s a public facing system, you’ll and producing usable reports can be a challenge. There want to consider the security of the system in its design are commercial tools such as SurveyMonkey, SurveyFrog, and operation. For our BSD Certification Group (BSDCG) Zoomerang, and others. If, as we did, you want to stay as website, we naturally chose a BSD operating system. much open source as possible, there are tools such as Fortunately, BSD systems have several different packages FreeOnlineSurveys, and phpESP. for webservers and administration tools. As with any We found that we could not use some of the ‘free’ website, it’s best to keep the operational requirements to tools since they provided only limited service such as a bare minimum. In Part II we noted that while we do have only 12 questions, 100 responses, and 10 days active. If a website, none of our certification test items (questions or your needs go beyond the limitations of these free tools, answers) are on that site. We wanted to make sure our site consider hosting your own survey with your own software. was always available, so we engaged with a group in Brazil We found phpESP to be easy to integrate into one of our to set up a mirror. We currently use rsync to synchronize BSD hosts. Survey creation and use was tolerable, but any changes we make on the primary system to the we found ways to automate the basic tasks using some synchronized secondary system. There is no cost and perl scripting and knowledge of SQL. This allowed us to relatively little overhead with this approach . rapidly create surveys with hundreds of questions, and since we hosted it ourselves, we could control the active Monitoring time and number of responders. If this is the way you want You should install intrusion detection and prevention / to go, keep in mind that you’ll need some technical know- monitoring tools such as Snort or Suricata. These tools how for databases, scripting, other technical tasks. 46 08/2011
- 2. What It Takes – Starting and Running an Open Source Certification Program, Part III Collaboration Open Source Certification Program with nothing but paper By now, you’ve got hundreds, maybe even hundreds of and pencil, but it very quickly becomes unsustainable. thousands of people using your software and there is a There will be hundreds, possibly thousands, of items lot of buzz. People need to communicate, and it’s your job and answers. And there will be plenty of errors and to enable them to do that in multiple ways. Fortunately, corrections. Computerizing your construction and delivery that’s what the Internet excels at- communication and are essential. collaboration. We set up several mailing lists with GNU The discussion in this section centers on creating and Mailman. We have one for our public discussions about maintaining your item collection within your own systems. our Certification Program and our announcements, If / when you migrate to an online test creation systems one for our private business discussions, one for our such as Schroeder Measurement Technologies (SMT) infrastructure team to keep up with configurations and your test creation, delivery, and scoring procedures will changes, and one for our Subject Matter Experts (SMEs). be completely different. However, the basic concepts will Mailman is easy to administer and has proven capable of remain the same. easily handling our workload. For real time discussion and collaboration, we use Internet Relay Chat (IRC). We have Item Collection and Management our #bsdcert channel set up on freenode.net. This also The BSDCG started out with item collections as just serves as a friendly chat forum for us as well. word processing documents, mostly in Open Office. With the rise of social networking we’ve taken These were easy to organize and exchange with our advantage of groups on LinkedIn and Facebook. These psychometrician. They were also easy to distribute to our groups allow us to immediately reach hundreds, even SMEs when we had question writing sessions. And while thousands of people with daily or weekly updates. It’s also these SME sessions generated a number of edits across where we can reach out to the community for advice and several documents, it wasn’t that much work to keep it all support our initiatives such as new surveys, new exams, straight. The problem was pushing out questions into an or collaboration on editing reports. actual exam. For additional collaboration, consider an open source document management system (also known as a content Form Creation management sytsem – CMS). There are some excellent Psychometricians and exam creators use special CMS systems in use such as Plone or Drupal. Some terminology when describing exam construction. A form of them are very easy to use. If you decide to manage is the name given to the collection of items that will be put your content in one of these systems, consider getting a onto an exam. Creating our exam forms required several support contract with a firm that specializes in supporting steps as outlined in this simplified view: that software. You will get timely notification of bugs and security holes- and hopefully fixes for your money. • Collecting items by knowledge domain. Knowledge domains and objectives were developed during the Candidate Registration Job Task Analysis. In this step, we are collecting All certification systems need a way to identify each actual items to be used on the exam. candidate with some form of identification number. • Ensuring that all the objectives are covered by at The BSDCG developed an on-line application to take least one item, and ensuring that only the highest the candidate information and give them a registration quality items get put onto an exam. This is done in number. This number is used to identify the candidate item writing sessions with Subject Matter Experts. within the BSDCG internal systems, not on any public • Drafting up a form by copying selected items from website. Registration numbers are also used when each domain and randomizing the order on the printing exam results and certificates. form. Eventually we plan to have another web based system to • Double checking to ensure the percentages of enable employers to verify whether a prospective employee all items for each domain are correct. The exam is BSDCG certified. For now, this function is performed objectives document contains the percentage for manually, via email to [email protected]. each domain established by the psychometrician. • All forms are vetted and approved by the Test Construction and Delivery psychometrician before use. This usually requires a Test construction and delivery will be your biggest beta period where items are tested for psychometric challenge over time. It is possible to start and run an correctness. www.bsdmag.org 47
- 3. LET’S TALK This results in the actual form to be used by the exam • A copy of the exam form is printed and stapled candidate. That form is now ready for packaging and together by the exam shipper. Each BSDA exam is delivery. about 12 pages long. • Immediately after printing, the exam form is sealed Exam Survey with an adhesive tab. At the end of every exam form we include a short survey. • The first page of the exam form consists of a Non- The exam instructions ask the candidate to fill out the Disclosure Agreement indicating that, by opening the survey before returning the exam materials back to the seal, the candidate agrees not to discuss any aspects proctor. This survey is about the exam session itself of the exam. The candidate is required to sign the – Was the room quiet? Was it too hot or too cold? Did agreement before continuing with the exam. the proctor explain the procedures clearly? Were there • The sealed exam form, a single answer sheet, and any disruptions during the exam, etc. Getting personal one blank colored sheet of paper are put into a feedback about the exam session is important, and is numbered manilla envelope. The colored sheet is required for accrediting your Certification Program. Plus, available for the candidate to write down any issues it gives the candidate a chance to communicate directly they have with any question. It gives them the chance with the exam program administrators. With all BSDCG to indicate that a question was unclear, if they thought exams, we read every survey and take note of every the answers were all wrong, or any similar issue. comment. • The envelope is sealed. Special BSDCG packing tape is placed across the seal to prevent tampering. Only Answer Sheets the exam candidate is permitted to open the envelope You will want to discuss scoring the results of the exam seal, and the exam seal. Once they are finished with your psychometrician. To be effective and reliable, with the exam, they are required to place the exam the scoring must be independent of the exam delivery form, answer sheet, and colored paper back into the process. Your psychometrician may want to use specific envelope and reseal the envelope, writing their name equipment or have you engage with a separate company across the seal. to score the exam and report the results. • We maintain an internal spreadsheet that contains In our case, we engaged Dr. Sandra Dolan to score the the candidate name, exam form number, and the exam for us as part of her contract. She uses a scoring envelope number as well as the date and location of machine that reads the Pearson NCS MM4521-3 Answer the exam and the name of the proctor. These details Sheet. These forms, familiar to almost anyone who has are important if it later becomes obvious that there ever taken a certification exam, are commonly known as were problems with the exam session. bubble sheets. Each item has a row of tiny circles (A, B, C, • The same procedures above are performed for D, E) for answer marks. The form also has a place for the all candidates at a particular exam session. The candidate name, birthdate, and some identification codes. collection of sealed exam packs are then boxed up Note that these forms are copyrighted by Pearson NCS and mailed directly to the proctor. Included in the and must be purchased, not copied, for use. box are the proctor directions and a list of candidates expected for that session. Packaging and Test Delivery Paper and pencil exams have very specific procedures If you are mailing internationally, you’ll need to allow plenty that must be followed in order to protect your exam of time for the materials to reach the proctor before the content. You can’t just send a bunch of copies in an exam. Six weeks lead time is highly recommended. envelope and mail it to the proctor. If you do that, you have no way of knowing if the contents were copied or stolen Proctoring the Exam Session by another candidate, the proctor, or someone else. In On the appointed day, the proctor arrives at the exam fact, when followed, the procedures below permit only the location, and sets up the exam session room. Proctors exam shipper who has already signed a Non-Disclosure are required to check the identification of each candidate Agreement and the candidate themselves to actually view (a government issued photo ID is required). The candidate the exam. No one else, including the proctor, will be able is given their exam packet and told to have a seat until the to view the actual exam with these procedures. start of the exam. Here are the packaging procedures we use in the Once the proctor begins the exam session, no further BSDCG: candidates are allowed into the exam session. The 48 08/2011
- 4. What It Takes – Starting and Running an Open Source Certification Program, Part III proctor reads the exam instructions, verifies that everyone • Registrations – How many people are interested in is ready, and starts the clock. your Certification Program enough to actually register At the end of the exam, the proctor (as noted above) to take the exam? While more registrations are directs the candidates to place the exam form, answer better overall, there are several reasons why people sheet, and colored paper back into the envelope and will register and then not actually take the exam – reseal it, writing their name across the seal. Sealed exam lack of time, distance from a test center, a change packs are boxed back up and returned to the BSDCG for in personal goals, and others. On the other hand, a scoring. continual stream of registrations shows that there is still continued interest. Scoring and Reporting • Tests Given – How many tests sessions have been The exam packs are returned to the exam shipper who scheduled and completed? This is a concrete number checks each pack for completeness, reads the sign- of people who have registered, paid, shown up and in sheets, the exam survey, and all notes which are actually taken the exam and is a very useful metric. recorded. A copy of the answer sheet is archived and • Candidates Pass / Fail – The number of people who the orginals are sent to the psychometrician. The actual pass and the number of people who fail the exam are exams are also archived and securely stored at the very useful indicators about the quality and validity of BSDCG home office. the exam. Consider the extremes. If everyone failed, Scoring the exam is performed by the psychometrician the exam wasn’t focused on the objectives, or the to ensure proper separation of duties for exam delivery items were somehow distorted. If everyone passed, and scoring. The scoring activity itself is automated. the items were too easy. Your psychometrician will Answer sheets are fed into a special reader (Scantron help determine the best pass / fail ration for your iNSIGHT 4ES, OpScan 4ES, or similar reader) which Program. uses special software. • Usefulness – How many candidates took the exam The raw answers for each scan are captured and are fed and were able to advance their careers with this new into additional software used for psychometric analysis. certificate? If candidates can show their certificate to This is an important aspect of the scoring procedure. a prospective employer who recognizes the value of During this analysis, the psychometrician can compare the Certification Program, you are definitely on the the results from a single test to other recent exam scores. right track. For example, the analysis can reveal whether an item • Buzz – Does your Certification Program have industry is consistently being answered incorrectly – indicating buzz? Are media pundits, bloggers, and technology a poorly worded item or answer set. It can also reveal journalists aware of your program? This is a very if there are too many commonalities in answers among useful, if ephemeral metric. Enjoy it while it lasts. a group of candidates in one exam session – indicating possible cheating. This analysis is vital to the integrity of Conclusion the Certification Program. You’ve built your software. You’ve built a community of Once the items are scored it is time to communicate dedicated People. You’ve set up your Procedures and your with the candidate whether they passed or failed. A PDF Technology as you’ve built your Certification Program. report indicating whether they have passed or failed is Now all you have to do is conquer the technology world. generated and sent to the candidate. If the candidate Carpe Diem! passed, a certificate is printed, an individual foil seal is applied to the certificate, the remaining third signature is added, an ID card and welcome letter are printed, and all are mailed to the candidate. JIM BROWN Assessing the Program’s Effectiveness Jim Brown has worked in the computer industry with continuous How will you know if your Certification Program is Unix involvement in development or administration since the successful? Money is one obvious answer, but not the early 1980s. His experience includes applications, systems only one. You’ll be successful if you can demonstrate and database programming, in a variety of languages. One of that your program is effective in supporting your most the founders of the BSD Certi�cation Group, he is helping to excellent software. Below are some metrics that may be develop the BSD Professional certi�cation. He currently lives in useful in tracking effectiveness: Northwest Arkansas, USA. www.bsdmag.org 49